eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a n+1 by n matrix of integers. I want to find a linear combination of the rows that reduces to zero mod 2. How would I do this in python? I could write gaussian elimination on my own, I feel there ought to be a way to do this using numpy or other library without writing it from scratch. Example: I have the matrix [1, 3, 0] [1, 1, 0] [1, 0, 1] [0, 1, 5] The function should return [1, 0, 1, 1]. Because that linear combination yields [2,4,6] = [0,0,0] mod 2.

Answer: You can use galois: from galois import GF2 import numpy as np A = [[1, 3, 0], [1, 1, 0], [1, 0, 1], [0, 1, 5]] A = GF2(np.array(A).T % 2) print(A.null_space())  It gives: [[1 0 1 1] [0 1 1 1]]  The rows are a basis of the null space of the matrix over the field F_2.
Question: I've got a very large pandas df that has the fields group, id_a, id_b and score, ordered by score so the highest is at the top. There is a row for every possible combination of id_a and id_b. I want to extract rows so that there is only one row per id_a and id_b, which reflects the highest score possible without repeating IDs. Showing an example of what this might look like - the resulting df has 3 rows, with all ids in id_a and id_b appearing once each. In the case of A2/B2 and A1/B1, the row with the best score for both IDs has been used. In the case of A3, the best score related to a row with B1, which had already been used, so the next best score combined with B3 is used. Input table  Desired result  To achieve this, I've got a loop iterating through the original df. This is incredibly slow with a large dataset. I've tried coming up with alternatives but I am struggling, for e.g.:  Once a row is identified for a pair of IDs, I could remove those IDs from the original df, but I'm not sure how to do this without restarting the loop I could split things by group (the example only has 1 group but there would be lots more, with IDs unique across groups) - however this doesn't seem to save time  Can anybody offer any other approaches? Thank you! import pandas as pd # Create sample df group = [1, 1, 1, 1, 1, 1, 1, 1, 1] id_a = ['A2', 'A1', 'A3', 'A3', 'A2', 'A1', 'A1', 'A2', 'A3'] id_b = ['B2', 'B1', 'B1', 'B3', 'B1', 'B2', 'B3', 'B3', 'B2'] score = [0.99, 0.98, 0.97, 0.96, 0.93, 0.5, 0.41, 0.4, 0.2] df = pd.DataFrame({'group': group, 'id_a': id_a, 'id_b': id_b, 'score': score}) result = pd.DataFrame(columns=df.columns) # Extract required rows for i, row in df.iterrows(): if len(result) == 0: result = row.to_frame().T else: if ((row['id_a'] in result['id_a'].tolist()) or (row['id_b'] in result['id_b'].tolist())): continue else: result = pd.concat([result, row.to_frame().T[result.columns]]) 

Answer: This looks to me line a linear_sum_assignment problem (i.e. maximizing the sum of score for unique pairs of id_a/id_b), you could use a pivot per group to do this: from scipy.optimize import linear_sum_assignment out = {} for group, g in (df.pivot(index=['group', 'id_a'], columns='id_b', values='score') .groupby(level=0)): idx, col = linear_sum_assignment(g, maximize=True) out[group] = pd.DataFrame({'id_a': g.index.get_level_values(1)[idx], 'id_b': g.columns[col], 'score': g.values[idx, col]}) out = pd.concat(out, names=['group']).reset_index(0)  Alternative code (does the same thing with a different order of the steps). This way is preferred if you don't have the same id_a/id_b across groups (this avoids building a large pivoted intermediate): from scipy.optimize import linear_sum_assignment import numpy as np out = [] for group, g in df.sort_values(by=['id_a', 'id_b']).groupby('group'): tmp = g.pivot(index='id_a', columns='id_b', values='score') idx, col = linear_sum_assignment(tmp, maximize=True) out.append(g.iloc[np.ravel_multi_index((idx, col), tmp.shape)]) out = pd.concat(out)  Output:  group id_a id_b score 0 1 A1 B1 0.98 1 1 A2 B2 0.99 2 1 A3 B3 0.96
Question: I read the Python3 references for the Import statement here, in which it said:  The from form uses a slightly more complex process: find the module specified in the from clause, loading and initializing it if necessary; for each of the identifiers specified in the import clauses: check if the imported module has an attribute by that name if not, attempt to import a submodule with that name and then check the imported module again for that attribute if the attribute is not found, ImportError is raised. otherwise, a reference to that value is stored in the local namespace, using the name in the as clause if it is present, otherwise using the attribute name  For the quoted block in above, I cannot figure out what does it really say about the form: "from A import B". In my understanding, in that case, if B is not an attribute of A, then the process will search for a submodule named as "B", if that is found, the process would search an attribute named "B" in B submodule(from "then check the imported module again for that attribute"), is my understanding to that correct or not ? Need your help.

Answer: It's checking the first module again, not the submodule. So for example, from foo import bar will find the foo module and check it for a bar attribute. If there's no such attribute there, it will attempt to import a foo.bar submodule. If this new import succeeds, it should ordinarily result in the submodule getting set as a bar attribute on foo. The first import will then check foo again for the bar attribute it expects to now be there. (The description in the docs leaves out one last step - if there's still no bar attribute on foo, then the import will try to retrieve foo.bar directly from sys.modules. This matters in some circular import cases, due to how the initialization order works out in circular imports.)
Question: Recently I have installed Jupyter Notebook on my Mac for didactical use and I've noticed a problem: doing list =  Jupyter doesn't let me type the left square bracket, and if I type that in a new line it allows me to type, for example: []  I've just tried to reinstall completely Jupyter from the prompt using Homebrew and it doesn't worked. How can I do?

Answer: This is a known issue: https://github.com/jupyterlab/jupyterlab/issues/15744 You can do: Settings → Settings Editor → JSON Settings Editor → Keyboard Shortcuts → User → paste → save  paste: { "shortcuts": [ { "args": {}, "command": "inline-completer:next", "keys": [ "Alt ]" ], "selector": ".jp-mod-completer-enabled", "disabled": true }, { "args": {}, "command": "inline-completer:previous", "keys": [ "Alt [" ], "selector": ".jp-mod-completer-enabled", "disabled": true }, { "args": {}, "command": "inline-completer:invoke", "keys": [ "Alt \\" ], "selector": ".jp-mod-completer-enabled", "disabled": true } ] }  https://github.com/jupyterlab/jupyterlab/issues/15744#issuecomment-1999433752 As @everson commented please restart your notebook after changes.
Question: When using asyncio and aiohttp, faced this issue and was wondering if it was an occurance limited to VSC only? This one keeps leaving residues from the previous loops. while True: loop.run_until_complete(get_data())   This runs without any residual tasks. while True: asyncio.run(get_data())   session = aiohttp.ClientSession() async def fetch(url, params=None): async with semaphore: async with session.get(url, params=params) as response: return await response.json()  I tried moving the while loop inside the called method itself but still the same result. Also tried using semaphores with no better luck. I was trying to manage multiple rest api call methods using aiohttp from within a single loop but with this happening, guess I am stuck with having to use asyncio.run separately for each api related method. So the only way to avoid this right now is to keep closing and reopening separate loops? Could this possibly be a VSC related issue?  Edit: Tried something with Python 3.10.6 and VSCodium. I don't know if it's because this one shows tasks/loops instead of threadpool here but the same issue is not replicated. The increment issue still persists when using Python 3.10.6 with VS Code. So could this be either 1) VSC issue as suspected or 2) just that the call stacks are being treated different between the two IDEs? while True: asyncio.run(get_data())   Another thing to note is that the ThreadPoolExecutor stacking when using VSC, stops exactly at the 20th stack: ThreadPoolExecutor-0_19. Then no more increments and no crashes. 

Answer: This worked. Still not sure as to the different results between VSC and VSCodium with the initial code but nonetheless, for anyone else who stumbles through. async def fetch(url, params=None): if not self.session: self.session = aiohttp.ClientSession() async with semaphore: async with session.get(url, params=params) as response: return await response.json() async def get_data(self): json_response = await self.fetch("https://jsonplaceholder.typicode.com/todos/1") while True: asyncio.run(get_data())
Question: I am trying to use the packages scrapy and great_expectations within the same virtual environment. There seems to be an issue with the compatibility between the two packages, depending on the order in which I import them in. Example:  I created a virtual environment and pip installed the latest version of each package.  Does work: import great_expectations import scrapy print("done")  Does not work: import scrapy import great_expectations print("done")  Error: Traceback (most recent call last): File "/Users/grant/vs_code_projects/grants_projects/test_environment.py", line 2, in <module> import great_expectations File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/__init__.py", line 32, in <module> register_core_expectations() File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/registry.py", line 187, in register_core_expectations from great_expectations.expectations import core # noqa: F401 File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/__init__.py", line 1, in <module> from .expect_column_distinct_values_to_be_in_set import ( File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/expect_column_distinct_values_to_be_in_set.py", line 12, in <module> from great_expectations.expectations.expectation import ( File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 2350, in <module> class BatchExpectation(Expectation, ABC): File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 287, in __new__ newclass._register_renderer_functions() File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 369, in _register_renderer_functions attr_obj: Callable = getattr(cls, candidate_renderer_fn_name) AttributeError: __provides__  Edit:  I have tried all major versions of Scrapy (current version, 2.0.0, 1.0.0) with all major versions of Great Expectations and it is the same result. 

Answer: This has been fixed as of 0.18.13 https://github.com/great-expectations/great_expectations/releases/tag/0.18.13
Question: I have a very large Pandas Dataframe ~9 million records, 56 columns, which I'm trying to load into a MSSQL table, using Dataframe.to_sql(). Importing the whole Dataframe in one statement often leads to errors, relating to memory. To cope with this, I'm looping through the Dataframe in batches of 100K rows, and importing a batch at a time. This way I no longer get any errors, but the code slows down dramatically after about 5.8 million records. The code I'm using: maxrow = df.shape[0] stepsize = 100000 for i in range(0, maxrow, stepsize): batchstart = datetime.datetime.now() if i == 0: if_exists = 'replace' else: if_exists = 'append' df_import = df.iloc[i:i+stepsize] df_import.to_sql('tablename', engine, schema='tmp', if_exists=if_exists, index=False, dtype=dtypes )  I've timed the batches, and there is a clear breaking point in speed:  These results are basically the same for batches of 50k, 100k and 200k rows. it takes about 40 minutes to upload 6 million records, and another 2 hours and 20 minutes to upload the next 3 million. My thinking was that it was either due to the size of the MSSQL table, or something being cached/saved after each upload. Because of that I've tried pushing the Dataframe to two different tables. I've also tried something like expunge_all() on the SQLALchemy session, after each upload. Both to no effect. Manually stopping imports after 5 million records and restarting from 5 million with a new engine object also hasn't helped. I'm all out of ideas what might be the cause of the process slowing down so drastically, and would really appreciate help. UPDATE As a last resort I've reversed the loop, uploading parts of the Dataframe starting at the highest index, looping down. This has basically reversed the times per batch. So it seems it is the data itself that is different/bigger further down the Dataframe. Not the connection being overloaded or the SQL table getting to large.  Thanks to everyone trying to help, but it seems I need to go through the data to see what causes this.

Answer: I've found a solution, which might be useful for someone else looking to speed up a slow(ing) Dataframe.to_sql() operation, and has already tried things like the chunksizes and setting up the SQLALchemy connection with fast_executemany=True. Not sure what mechanism causes this change in performance, but it's worked for me, and I haven't seen it mentioned anywhere else. So I hope it helps someone else. Sorting the original Dataframe with DataFrame.sort_values() has cut the time needed by roughly 2/3.  It seems to be very important on which columns the values are sorted though. The speed is best when the Dataframe is sorted by a column which is unique and acts like a nested/clustered sort. Sorting by this column implicitly sorts by about 20 columns. It's sorting addresses, which also sorts street, postal code, city, region etc. That also means that a lot of rows within a batch share a lot of values within the rows. When the Dataframe is sorted by another column which has no connection to other columns it actually slows down the Dataframe.to_sql() quite dramatically.
Question: I've been using exchangelib library in python for a long time now to access emails in an email account. It's been working amazing. Out of the blue today, I get an error saying, KeyError: 'folders' During handling of the above exception, another exception occured: exchangelib.errors.ErrorAccessDenied: Not allowed to access Non IPM folder.  The line of code this is happening on is right here. msg_folder= my_account.root / 'Top of Information Store' / 'my_subfolder'  Like I mentioned, this has been working great for over a year now. Double checked that access for the microsoft application is correct, and it is. It also looks like the latest release of exchangelib was 3/8/24 and it's been working since then, so it can't be that. The other weird thing, is a separate script is able to access messages in the inbox, it's just this sibling folder to the inbox that is throwing the error. I'm not finding anything on this error though. Any ideas on how to fix this?

Answer: This is caused by a recent change in O365. We may be able to find a fix for it in exchangelib. Until then, a workaround is to navigate to folders using double slashes: msg_folder = my_account.root // 'Top of Information Store' // 'my_subfolder'  This works by not collecting the full folder hierarchy first and navigating the client-side folder cache, but rather asking the server for the specific child folder each time we reach a new // level. UPDATE: fix provided in https://github.com/ecederstrand/exchangelib/issues/1290
Question: I'm working on an AI project that involves object detection and action recognition on roughly 30 minute videos. My pipeline is the following:  determine crops using object detection model extract crops using Python and write them to disk as individual images for each frame. use action recognition model by inputting a series of the crops.  The models are fast but actual writing of the crops to disk is slow. Sure, using an SSD would speed it up but I'm sure ffmpeg would greatly speed it up. Some of the challenges with the crops:  the output size is always 128x128 the input size is variable the crop moves on every frame  My process for extracting crops is simple using cv2.imwrite(output_crop_path, crop) in a for loop. I've done experiments trying to use sndcommand and filter_complex. I tried this https://stackoverflow.com/a/67508233/4447761 but it outputs an image with black below the crop and the image gets wrapped around on the x axis. 

Answer: Don't store the pictures. Store just the sequence of bounding boxes. Then, for whatever you wanted to do with that mountain of individual images, instead decode the video and read the sequence of boxes, and take your crops out of it like that, on the fly. I'd recommend using PyAV for video reading. It gives you the presentation timestamps reliably. You shouldn't even need them, but having them along in the file can be very helpful for debugging. Maybe you want to use pandas to write and read such a file. CSV is a popular format. Another advantage of keeping the boxes like that: you can work on this sequence, maybe smooth it, or mix in the results of an optical tracker (sticks like glue but also creeps over time).  If you really do need to write a video, then write a video. Individual frames as files are a serious burden on most file systems, let alone file managers. Again I would recommend PyAV to write a video. It gives you all kinds of control over how to do it, and the basic code isn't all that much. PyAV comes with examples.
Question: T = TypeVar("T", bound=Union[str, int]) def connect_lists(list_1: list[T], list_2: list[T]) -> list[T]: out: list[T] = [] out.extend(list_1) out.extend(list_2) return out connect_lists([1, 2], ["a", "b"])  mypy:  error: Cannot infer type argument 1 of "connect_lists" [misc]  T = TypeVar("T", bound=Union[str, int]) def connect_lists(list_1: Iterable[T], list_2: Iterable[T]) -> list[T]: out: list[T] = [] out.extend(list_1) out.extend(list_2) return out connect_lists([1, 2], ["a", "b"])  Now mypy doesn't raise an error. What is the difference between List and Iterable in this case?

Answer: Iterable is covariant - an Iterable[int] is also an Iterable[int|str]. list is not covariant - a list[int] is not a list[int|str], because you can add strings to a list[int|str], which you can't do with a list[int]. mypy infers the types of [1, 2] and ["a", "b"] as list[int] and list[str] respectively. With the first definition of connect_objects, there is no choice of T that will make the call valid. But with the second definition, a list[int] is an Iterable[int], which is an Iterable[int|str], and a list[str] is similarly also an Iterable[int|str], so T is inferred as int|str. I don't think there's actually a spec yet for how type inference works. There may never be such a spec. Future mypy versions might perform this inference differently, for example, performing context-sensitive inference to infer a type of list[int|str] for both input lists, making the first version of the code pass type checking.
Question: Somebody asked me a question and I honestly didn't try it before, So it was interesting to know what exactly happens when we name a module __main__.py. So I named a module __main__.py, imported it in another file with name test.py. Surprisingly when I tried to run test.py it prints nothing and none of the functions of __main__.py are available in test.py. Here is the contents of these files : Here is the contents of __main__.py : def add(a,b): result = a+b return result print(__name__) if __name__=='__main__': print(add(1,2))  Here is the contents of test.py : import __main__  Why isn't the print statement from __main__.py reached? Although when I rename the __main__.py with some other name such as func.py, the program runs correctly and the line that prints module's name works fine.

Answer: When you run python test.py, the test.py module itself is already present in sys.modules with the key "__main__" (see top-level code environment in the docs). The import __main__ will just return this existing cache hit, so the presence of a __main__.py file on sys.path is irrelevant. These modifications to test.py will prove that point: import sys print(sys.modules['__main__']) # it's already in there before the import import __main__ print(__main__.__file__) # this should be the abspath of test.py
Question: I am trying to compare two lists of odds from two bookmakers. They look like this: List1 = ['2.66', '3.79', '1.88', '1.61', '2.51', '1.29', '2.29', '2.56', '3.16', '2.05', '2.95', '2.64', '2.26', '3.17', '2.64', '2.25'] List2 = ['2.70', '4.40', '1.87', '1.56', '2.50', '1.26', '2.33', '2.60', '3.20', '2.04', '3.00', '2.65', '2.25', '3.20', '2.65', '2.22']  I need to merge them and get the highest odds. I already did this with numpy: numpy.array([List1, List2]).astype(float).max(axis = 0) FinalList = [2.7 4.4 1.88 1.61 2.51 1.29 2.33 2.6 3.2 2.05 3.2 2.65 2.26 3.2 2.65 2.25]  The problem is that I can't know to which list each index belongs to. In this example what I need to get is: NamesLists = [List2, List2, List1, List1, List1, List1, List2, List2, List2, List1, List2, List2, List1, List2, List2, List1]  But I really have no idea how to do this.

Answer: You can combine argmax and take_along_axis: import numpy List1 = ['2.66', '3.79', '1.88', '1.61', '2.51', '1.29', '2.29', '2.56', '3.16', '2.05', '2.95', '2.64', '2.26', '3.17', '2.64', '2.25'] List2 = ['2.70', '4.40', '1.87', '1.56', '2.50', '1.26', '2.33', '2.60', '3.20', '2.04', '3.00', '2.65', '2.25', '3.20', '2.65', '2.22'] tmp = numpy.array([List1, List2]).astype(float) idx = tmp.argmax(axis=0) FinalList = numpy.take_along_axis(tmp, idx[None], axis=0)[0] # or: FinalList = tmp[idx[None], numpy.arange(tmp.shape[1])][0] # array([2.7 , 4.4 , 1.88, 1.61, 2.51, 1.29, 2.33, 2.6 , 3.2 , 2.05, 3. , # 2.65, 2.26, 3.2 , 2.65, 2.25]) NamesLists = numpy.array(['List1', 'List2'])[idx] # array(['List2', 'List2', 'List1', 'List1', 'List1', 'List1', 'List2', # 'List2', 'List2', 'List1', 'List2', 'List2', 'List1', 'List2', # 'List2', 'List1'], dtype='<U5')  Note that idx is of the form: array([1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0])  which might be easier to use than ['List2', 'List2', 'List1', ...]
Question: We are deploying an open source application based on numpy that includes libopenblas.{cryptic string}.gfortran-win32.dll. It is part of the Python numpy package. This dll is over 27MB in size. I'm curious why it is so big and where I can find the source for it to see for myself. Ultimately I'd like to see if it can be more limited in size for my application. Thanks

Answer: OpenBLAS includes many optimized kernels for different CPU architectures and instruction sets. This is why the DLL is too large. Alternatives like BLIS and Intel's Math Kernel Library (MKL) exist. You can give a try to them if you consider. You can also build the library from the beginning and strip off the unnecessary parts while building, maybe.
Question: Say I have a class defined as follows: class MyList(list[int]): ...  I'm looking for a method that will return list[int] when I give it MyList, e.g.: >>> inspect.getsupers(MyList) [list[int]]  The trouble is that no such method exists, as far as I can find. There is inspect.getmro(...), but this only returns list, not list[int]. Is there anything that I can reuse or implement that will give me this, short of some magic involving inspect.getsource(...)? The context is that I'm trying to write a generic issubtype(t1, t2) method that works for arbitrary classes and GenericAliases. This is one issue I've run into in writing this method that I'm not sure how to solve.

Answer: Aha! This information is stored in .__orig_bases__. So: >>> MyList.__orig_bases__ (list[int],)  And on 3.12+, the canonical way would be to use: >>> import types >>> types.get_original_bases(MyList) (list[int],)