eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm using psycopg2 library to connection to my postgresql database. Every time I want to execute any query, I make a make a new connection like this: import psycopg2 def run_query(query): with psycopg2.connect("dbname=test user=postgres") as connection: cursor = connection.cursor() cursor.execute(query) cursor.close()  But I think it's faster to make one connection for whole app execution like this: import psycopg2 connection = psycopg2.connect("dbname=test user=postgres") def run_query(query): cursor = connection.cursor() cursor.execute(query) cursor.close()  So which is better way to connect my database during all execution time on my app? I've tried both ways and both worked, but I want to know which is better and why.

Answer: You should strongly consider using a connection pool, as other answers have suggested, this will be less costly than creating a connection every time you query, as well as deal with workloads that one connection alone couldn't deal with. Create a file called something like mydb.py, and include the following: import psycopg2 import psycopg2.pool from contextlib import contextmanager dbpool = psycopg2.pool.ThreadedConnectionPool(host=<<YourHost>>, port=<<YourPort>>, dbname=<<YourDB>>, user=<<YourUser>>, password=<<YourPassword>>, ) @contextmanager def db_cursor(): conn = dbpool.getconn() try: with conn.cursor() as cur: yield cur conn.commit() # You can have multiple exception types here. # For example, if you wanted to specifically check for the # 23503 "FOREIGN KEY VIOLATION" error type, you could do: # except psycopg2.Error as e: # conn.rollback() # if e.pgcode = '23503': # raise KeyError(e.diag.message_primary) # else # raise Exception(e.pgcode) except: conn.rollback() raise finally: dbpool.putconn(conn)  This will allow you run queries as so: import mydb def myfunction(): with mydb.db_cursor() as cur: cur.execute("""Select * from blahblahblah...""")
Question: I have a function that will instantiate a huge array and do other things. I am running my code on TPUs so my memory is limited. How can I execute my function specifically on the CPU? If I do: y = jax.device_put(my_function(), device=jax.devices("cpu")[0])  I guess that my_function() is first executed on TPU and the result is put on CPU, which gives me memory error. and using jax.config.update('jax_platform_name', 'cpu') at the beginning of my code seems to have no effect. Also please note that I can't modify my_function() Thanks!

Answer: To directly specify the device on which a function should be executed, use the device argument of jax.jit. For example (using a GPU runtime because it's the accelerator I have access to at the moment): import jax gpu_device = jax.devices('gpu')[0] cpu_device = jax.devices('cpu')[0] def my_function(x): return x.sum() x = jax.numpy.arange(10) x_gpu = jax.jit(my_function, device=gpu_device)(x) print(x_gpu.device()) # gpu:0 x_cpu = jax.jit(my_function, device=cpu_device)(x) print(x_cpu.device()) # TFRT_CPU_0  This can also be controlled with the jax.default_device decorator around the call-site: with jax.default_device(cpu_device): print(jax.jit(my_function)(x).device()) # TFRT_CPU_0 with jax.default_device(gpu_device): print(jax.jit(my_function)(x).device()) # gpu:0
Question: In asynchronous queries, I want to get foreign key and many to many attributes of a model instance. In a simple example, I want to print university and courses for all instances of the model Student. models.py: from django.db import models class University(models.Model): name = models.CharField(max_length=64) class Course(models.Model): name = models.CharField(max_length=64) class Student(models.Model): name = models.CharField(max_length=64) university = models.ForeignKey(to=University, on_delete=models.CASCADE) courses = models.ManyToManyField(to=Course)  when I use this code (in django 4.1): import asyncio async def main(): async for student in Student.objects.all(): print(student.name) print(student.university.name) for course in student.courses.all(): print(course.name) asyncio.run(main())  I get the following error: django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.  How can I fix this error?

Answer: For getting foreign key and many to many attributes (for django 4.1 or higher) in async queries: Foreign key attribute: There are two options for getting foreign key attribute: 1- async for student in Student.objects.all(): university = await University.objects.aget(id=student.university_id) print(university.name)  2- async for student in Student.objects.select_related("university").all(): print(student.university.name)   Many to Many attribute: async for student in Student.objects.all(): async for course in student.courses.all(): print(course.name)
Question: I have some issue with using Fetch API JavaScript method when sending some simple formData like so: function register() { var formData = new FormData(); var textInputName = document.getElementById('textInputName'); var sexButtonActive = document.querySelector('#buttonsMW > .btn.active'); var imagesInput = document.getElementById('imagesInput'); formData.append('name', textInputName.value); if (sexButtonActive != null){ formData.append('sex', sexButtonActive.html()) } else { formData.append('sex', ""); } formData.append('images', imagesInput.files[0]); fetch('/user/register', { method: 'POST', data: formData, }) .then(response => response.json()); } document.querySelector("form").addEventListener("submit", register);  And on the server side (FastAPI): @app.post("/user/register", status_code=201) def register_user(name: str = Form(...), sex: str = Form(...), images: List[UploadFile] = Form(...)): try: print(name) print(sex) print(images) return "OK" except Exception as err: print(err) print(traceback.format_exc()) return "Error"  After clicking on the submit button I get Error 422: Unprocessable entity. So, if I'm trying to add header Content-Type: multipart/form-data, it also doesn't help cause I get another Error 400: Bad Request. I want to understand what I am doing wrong, and how to process formData without such errors?

Answer: The 422 error response body will contain an error message about which field(s) is missing or doesn’t match the expected format. Since you haven't provided that (please do so), my guess is that the error is triggered due to how you defined the images parameter in your endpoint. Since images is expected to be a List of File(s), you should instead define it using the File type instead of Form. For example: images: List[UploadFile] = File(...) ^^^^  When using UploadFile, you don't have to use File() in the default value of the parameter, meaning that the below should work as well: images: List[UploadFile]  Hence, the FastAPI endpoint should look similar to this: @app.post("/user/register") async def register_user(name: str = Form(...), images: List[UploadFile] = File(...)): pass  In the frontend, make sure to use the body (not data) parameter in the fetch() function to pass the FormData object (see example in MDN Web Docs). For instance: var nameInput = document.getElementById('nameInput'); var imagesInput = document.getElementById('imagesInput'); var formData = new FormData(); formData.append('name', nameInput.value); for (const file of imagesInput.files) formData.append('images', file); fetch('/user/register', { method: 'POST', body: formData, }) .then(response => { console.log(response); }) .catch(error => { console.error(error); });  Please have a look at this answer, as well as this answer, which provide working examples on how to upload multiple files and form data to a FastAPI backend, using Fetch API in the frontend. As for manually specifying the Content-Type when sending multipart/form-data, you don't have to (and shouldn't) do that, but rather let the browser set the Content-Type—please take a look at this answer for more details.
Question: I have a dataset like this data = {'weight': ['NaN',2,3,4,'NaN',6,7,8,9,'NaN',11,12,13,14,15], 'MI': ['NaN', 21, 19, 18, 'NaN',16,15,14,13,'NaN',11,10,9,8,7]} df = pd.DataFrame(data, index= ['group1', "gene1", "gene2", 'gene3', 'group2', "gene1", 'gene21', 'gene4', 'gene7', 'group3', 'gene2', 'gene10', 'gene3', 'gene43', 'gene1'])   I need to stack it to gene by group dataframe with MI values. If there are no gene values for particular group the imputated value should be 0.1. 'weights' column should be removed. The final dataframe should look like this 

Answer: You can use: m = df['weight'].ne('NaN') (df[m] .set_index((~m).cumsum()[m], append=True)['MI'] .unstack('weight', fill_value=0.1) .add_prefix('group') )  Variant with pivot: m = df['weight'].ne('NaN') (df.assign(col=(~m).cumsum()) .loc[m] .pivot(columns='col', values='MI') .fillna(0.1) .add_prefix('group') )  Output: weight group1 group2 group3 gene1 21 16 7 gene10 0.1 0.1 10 gene2 19 0.1 11 gene21 0.1 15 0.1 gene3 18 0.1 9 gene4 0.1 14 0.1 gene43 0.1 0.1 8 gene7 0.1 13 0.1  update: keeping the original group names and lexicographic gene order from natsort import natsorted m = df['weight'].ne('NaN') grp = df.index.to_series().mask(m).ffill()[m] out = (df[m] .set_index(grp, append=True)['MI'] .unstack(-1, fill_value=0.1) .loc[natsorted(df.index[m].unique())] ) print(out)  Output:  group1 group2 group3 gene1 21 16 7 gene2 19 0.1 11 gene3 18 0.1 9 gene4 0.1 14 0.1 gene7 0.1 13 0.1 gene10 0.1 0.1 10 gene21 0.1 15 0.1 gene43 0.1 0.1 8
Question: Is it possible to select a potentially non-existent column from a polars dataframe without exceptions (return a column with default values or null/None)? The behavior I really want can be shown in the example as follows: import polars as pl df1 = pl.DataFrame({"id": [1, 2, 3], "bar": ["sugar", "ham", "spam"]}) df2 = pl.DataFrame({"id": [4, 5, 6], "other": ["a", "b", "b"]}) df1.write_csv("df1.csv") df2.write_csv("df2.csv") df = pl.scan_csv("df*.csv").select(["id", "bar"]) res = df.collect()  Now, if I run the code above, will get an error since df2.csv does not contain column "bar". The result I want is - res is just the contents in df1.csv, which means the dataframe in df2.csv will not be selected due to no column "bar" in it.

Answer: I mean as already in the comment mentioned above this functionality doesn't exist in polars, but we can construct a function which would fullfil your needs import glob def scan_csv_with_columns(file: str, needed_colnames: list[str]) -> pl.LazyFrame: file_collector = [] for filename in glob.glob(file): df_scan = pl.scan_csv(filename) if (df_scan.columns == needed_colnames): file_collector.append(df_scan) df = pl.concat(file_collector, how="vertical") return(df) file = "df*.csv" needed_colnames = ["id", "bar"] df = scan_csv_with_columns(file, needed_colnames) df.collect() shape: (3, 2) ┌─────┬───────┐ │ id ┆ bar │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪═══════╡ │ 1 ┆ sugar │ │ 2 ┆ ham │ │ 3 ┆ spam │ └─────┴───────┘
Question: I'm working on a project which uses FastAPI. My router file looks like the following: # GET API Endpoint 1 @router.get("/project/{project_id}/{employee_id}") async def method_one( project_id: str, organization_id: str, session: AsyncSession = Depends(get_db) ): try: return await CustomController.method_one( session, project_id, employee_id ) except Exception as e: return custom_exception_handler(e) # GET API Endpoint 2 @router.get("/project/details/{project_id}") async def method_two( project_id: str, session: AsyncSession = Depends(get_db) ): try: return await CustomController.method_two( session=session, project_id=project_id ) except Exception as e: return custom_exception_handler(e) # GET API Endpoint 3 @router.get("/project/metadata/{project_id}") async def method_three( project_id: str, session: AsyncSession = Depends(get_db) ): try: return await CustomController.method_three( session=session, project_id=project_id ) except Exception as e: return custom_exception_handler(e)  The obvious expectation of workflow here is: when each of these API endpoints are triggered with their required path parameters, the controller method is executed, as defined in their body. However, for some strange reason, when API endpoints 2 and 3 are triggered, they are executing the controller method in endpoint 1, i.e., CustomController.method_one(). Upon adding some print() statements in the method method_one() of the router, I've observed that method_one() is being called when API endpoint 2 is called, while it is actually supposed to call method_two() in the router. Same is the case with API endpoint 3. I'm unable to understand why the method body of method_one() is getting executed, when API endpoints 2 and 3 are triggered. Am I missing out something on configuration, or something - can someone please correct me? Thanks!

Answer: In FastAPI, as described in this answer, because endpoints are evaluated in order (see FastAPI's about how order matters), it makes sure that the endpoint you defined first in your app—in this case, that is, /project/{project_id}/...—will be evaluated first. Hence, every time you call one of the other two endpoints, i.e., /project/details/... and /project/metadata/..., the first endpoint is triggered, using details or metadata as the project_id parameter. Solution Thus, you need to make sure that the other two endpoints are declared before the one for /project/{project_id}/.... For example: # GET API Endpoint 1 @router.get("/project/details/{project_id}") # ... # GET API Endpoint 2 @router.get("/project/metadata/{project_id}") # ... # GET API Endpoint 3 @router.get("/project/{project_id}/{employee_id}") # ...
Question: In a poetry project the local dependencies are installed in the ~/.cache/pypoetry/virtualenvs/ folder. Pyright in nvim is complaining that import package lines can't be resolved. What should I include into pyproject.toml? Or how to show pyright the path to the dependencies? Thanks My pyrightconfig.json looks like this: { "venvPath": ". /home/ajanb/.cache/pypoetry/virtualenvs/", "venv": "tools-configfactory-materialmodel-jnEEQvIP-py3.10" }  I found that I need to add this to the config file of neovim, can you help me to write it in .lua?  au FileType python let b:coc_root_patterns = ['.git', '.env', 'venv', '.venv', 'setup.cfg', 'setup.py', 'pyrightconfig.json'] 

Answer: I spent days troubleshooting this. In the end, the only thing that worked was including this in my pyproject.toml: [tool.pyright] venvPath = "/Users/user/Library/Caches/pypoetry/virtualenvs" venv = "bfrl-93mGb6aN-py3.11"  I'm also using this nvim plugin: poet-v I guess you could accomplish this through a proper LSP configuration, but I was just not familiar enough with lua and the lsp configuration to tackle that automatically.
Question: Is there an efficient way to get an array of boolean values that are in the n-th position in bitwise array in Python?  Create numpy array with values 0 or 1:  import numpy as np array = np.array( [ [1, 0, 1], [1, 1, 1], [0, 0, 1], ] )   Compress size by np.packbits:  pack_array = np.packbits(array, axis=1)   Expected result - some function that could get all values from n-th column from bitwise array. For example if I would like the second column I would like to get (the same as I would call array[:,1]):  array([0, 1, 0])  I have tried numba with the following function. It returns right results but it is very slow: import numpy as np from numba import njit @njit(nopython=True, fastmath=True) def getVector(packed, j): n = packed.shape[0] res = np.zeros(n, dtype=np.int32) for i in range(n): res[i] = bool(packed[i, j//8] & (128>>(j%8))) return res  How to test it? import numpy as np import time from numba import njit array = np.random.choice(a=[False, True], size=(100000000,15)) pack_array = np.packbits(array, axis=1) start = time.time() array[:,10] print('np array') print(time.time()-start) @njit(nopython=True, fastmath=True) def getVector(packed, j): n = packed.shape[0] res = np.zeros(n, dtype=np.int32) for i in range(n): res[i] = bool(packed[i, j//8] & (128>>(j%8))) return res # To initialize getVector(pack_array, 10) start = time.time() getVector(pack_array, 10) print('getVector') print(time.time()-start)  It returns: np array 0.00010132789611816406 getVector 0.15648770332336426 

Answer: Besides some micro-optimisations, I dont believe that there is much that can be optimised here. There are also a few small mistakes in your code:  @njit(nopython=True) is saying the same thing twice (the n in njit already stands for nopython mode.) simply @njit or @jit(nopython=True) should be used fastMath is for "cutting corners" when doing floating point math, since we are only working with integers and booleans, it can be safely removed because it does nothing for us here.  My updated code (seeing a meagre 40% perfomance increase on my machine): import numba as nb import numpy as np np.random.seed(0) array = np.random.choice(a=[False, True], size=(10000000,15)) pack_array = np.packbits(array, axis=1) @nb.njit(locals={'res': nb.boolean[:]}) def getVector(packed, j): n = packed.shape[0] res = np.zeros(n, dtype=nb.boolean) byte = j//8 bit = 128>>(j%8) for i in range(n): res[i] = bool(packed[i, byte] & bit) return res getVector(pack_array, 10)  In your answer, "res" is a list of 32 bit integers, by giving np.zeros() the numba (NOT numpy) boolean datatype, we can swap it to the more efficient booleans. This is where most of the perfomance improvement comes from. On my machine putting j_mod and j_flr outside of the loop had no noticable effect. But it did have an effect for the commenter @Michael Szczesny, so it might help you aswell. I would not try to use strides, which @Nick ODell is suggesting, because they can be quite dangerous if used incorrectly. (See the numpy documentation). edit: I have made a few small changes that were suggested by Michael. (Thanks)
Question: Currently, I am validating the table schema with expect_table_columns_to_match_set by feeding in a list of columns. However, I want to validate the schema associated with each column such as string. The only available Great Expectations rule expect_column_values_to_be_of_type has to be written for each column name and also creates redundancy by repeating the column names. Is there any rule that I am missing that I can validate both the name and the schema at the same time? For exmaple, given column a: string, b: int, c: boolean, I want to pass that whole info into one function instead of having to break it into [a,b,c] and validating [a], string` separately for each column. Ideally, it will be something like expect_column_schmea([(column_name_a, column_type_a), (column_name_b, column_type_b)]

Answer: You can use expect_column_values_to_match_json_schema (or regex / pattern - depending on what you are more comfortable with). Here is the list of expectations that are possible to use. With expect_column_values_to_match_json_schema you can define your schema in a json format: schema = { "column_name_a": {"type": "string"}, "column_name_b": {"type": "integer"}, "column_name_c": {"type": "boolean"}, }  Create a new ExpectColumnValuesToMatchSchema instance (import for that was from great_expectations.expectations.core.expect_column_values_to_match_schema import ( ExpectColumnValuesToMatchSchema, )): expectation = ExpectColumnValuesToMatchSchema(schema=schema) And finally validate it to get your results: `result = expectation.validate(dataset)! You will get a ExpectationSuiteValidationResult as a return and can accordingly check whether the columns you provided match / do not match the schema!
Question: Since mid 2022 it is now possible to get rid of setup.py, setup.cfg in favor of pyproject.toml. Editable installs work with recent versions of setuptools and pip and even the official packaging tutorial switched away from setup.py to pyproject.toml. However, documentation regarding requirements.txt seems to be have been also removed, and I wonder where to put the pinned requirements now? As a refresher: It used to be common practice to put the dependencies (without version pinning) in setup.py avoiding issues when this package gets installed with other packages needing the same dependencies but with conflicting version requirements. For packaging libraries a setup.py was usually sufficient. For deployments (i.e. non libraries) you usually also provided a requirements.txt with version-pinned dependencies. So you don't accidentally get the latest and greatest but the exact versions of dependencies that that package has been tested with. So my question is, did anything change? Do you still put the pinned requirements in the requirements.txt when used together with pyproject.toml? Or is there an extra section for that in pyproject.toml? Is there some documentation on that somewhere?

Answer: Quoting myself from here  My current assumption is: [...] you put your (mostly unpinned) dependencies to pyproject.toml instead of setup.py, so you library can be installed as a dependency of something else without causing much troubles because of issues resolving version constraints.   On top of that, for "deployable applications" (for lack of a better term), you still want to maintain a separate requirements.txt with exact version pinning.  Which has been confirmed by a Python Packaging Authority (PyPA) member and clarification of PyPA's recommendations should be updated accordingly at some point.
Question: I would like to get the text inside this data structure that is outputted via GPT3 OpenAI. I'm using Python. When I print the object I get: <OpenAIObject text_completion id=cmpl-6F7ScZDu2UKKJGPXTiTPNKgfrikZ at 0x7f7648cacef0> JSON: { "choices": [ { "finish_reason": "stop", "index": 0, "logprobs": null, "text": "\nWhat was Malcolm X's original name?\nMalcolm X's original name was Malcolm Little.\n\nWhere was Malcolm X born?\nMalcolm X was born in Omaha, Nebraska.\n\nWhat was the profession of Malcolm X's father?\nMalcolm X's father was a Baptist minister.\n\nWhat did Malcolm X do after he stopped attending school?\nMalcolm X became involved in petty criminal activities." } ], "created": 1669061618, "id": "cmpl-6F7ScZDu2gJJHKZSPXTiTPNKgfrikZ", "model": "text-davinci-002", "object": "text_completion", "usage": { "completion_tokens": 86, "prompt_tokens": 1200, "total_tokens": 1286 } }  How do I get the 'text' component of this? For example, if this object is called: qa ... I can output qa['choices']  And I get the same items as above... but adding a .text or ['text'] to this does not do it, and gets an error. But not sure how to isolate the 'text' I've read the docs, but cannot find this... https://beta.openai.com/docs/api-reference/files/delete?lang=python

Answer: x = {&quot;choices&quot;: [{&quot;finish_reason&quot;: &quot;length&quot;, &quot;text&quot;: &quot;, everyone, and welcome to the first installment of the new opening&quot;}], } text = x['choices'][0]['text'] print(text) # , everyone, and welcome to the first installment of the new opening