eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've been on Python 3.8 for quite some time. I usually type hint with the convention: from typing import List, Union some_stuff: List[Union[int, str, float, List[str]]] = [98, "Fido", -34.925, ["Phantom", "Tollbooth"]]  I understand that with python 3.9 or greater you can type hint lists and collectible like: some_ints: list[int] If you have a union of types that a variable can occupy do you still need to import the Union class? like so: from typing import Union some_stuff: list[Union[int, str, float, list[str]]] = [98, "Fido", -34.925, ["Phantom", "Tollbooth"]]  Or is there a new convention for Union type hinting as well?

Answer: PEP 604, which was implemented in Python 3.10 allows Union types to be formed using the | operator (which has existed with exclusively numeric meanings for most of Python's history). So in sufficiently recent versions of Python, you can write your type hint like this, with no imports required from typing: some_stuff: list[int|str|float|list[str]] = [98, "Fido", -34.925, ["Phantom", "Tollbooth"]]  While that specifically answers your question, I'd note that it's a little suspicious that your example data exactly matches the order of the types you're listing in the Union. If you are always going to have exactly one of each kind of data (rather than some unknown mixture of them), you probably want to be type hinting the value differently, e.g. with tuple[int, str, float, list[str]] which specifies that there will be one value of each, in that order. You might need to use an actual tuple for the value, I'm not sure there's a way to hint a fixed pattern of values within a list object.
Question: As the title suggests, I am trying to use count() with a find() on a collection but it keeps throwing the error AttributeError: 'Cursor' object has no attribute 'count'. For reference, I went through this question but count_documents() seems to be tehre for colelctions themselves, and not cursors. The other option mentioned was len(list(cursor)) but I can't use that as it consumes the cursor itself (can't iterate over it after this). I went through a couple more answers, but these seem to be the main ways out. Moreover, my pymongo version is 4.3.3 which can't be changed due to some restrictions. Is there any operation I can perform directly on Cursor which doesn't consume it? Sample code def temp(col): return col.find().count() print(temp(collection))  Thanks!

Answer: list() will exhaust the cursor, but save its ouput to a variable and you can access it multiple times, e.g. records = list(col.find()) num_records = len(records) for record in records: # do stuff
Question: Something has substantially changed with the way that line profiling Cython works, such that previous answers no longer work. I am not sure if something subtle has changed, or if it is simply totally broken. For instance, here is a very highly upvoted question about this from about 8 years ago. The notebook in there no longer seems to work, even with the updates referenced in the post. For instance, here is a new version of the notebook incorporating the updates suggested in the original post: https://nbviewer.org/gist/battaglia01/f138f6b85235a530f7f62f5af5a002f0?flush_cache=true The output of line_profiler to profiling that Cython function is simply Timer unit: 1e-09 s  with no line-by-line infomration at all. I'm making a new question about this because other comments I've seen on the site all seem to reference these older answers, and they all seem to be broken. If anyone even has the beginnings of a starting point it would be much appreciated - either for the Jupyter notebook, or with something built using cythonize. My notes on what I've tried at least on the Jupyter side are in that notebook. Is there any way to get this to work?

Answer: The current status is: line_profiler v4 and Cython don't get on (for reasons that haven't yet been diagnosed but could be on either end). The line_profiler tests run as part of Cython's CI test-suite have line_profiler pinned to <4. It obviously isn't the long-term plan to leave this version pin, but if you need to it work now then do the same! The most recent run on the master branch as today used line_profiler v3.5.1 and this passed. Relevant issue: https://github.com/cython/cython/issues/5141
Question: I am trying to create a application which checks for sentence similarity. .exe file got created. I get the below error message while executing .exe file after giving required inputs. Error Message The 'tqdm>=4.27' distribution was not found and is required by this application. Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git main  spec file # -*- mode: python ; coding: utf-8 -*- block_cipher = None a = Analysis(['render_ui.py'], pathex=[], binaries=[], datas=[('Config\\favicon.ico',"."), ('Config\\miniLM.sav',".")], hiddenimports=['sklearn.metrics._pairwise_distances_reduction._datasets_pair', 'sklearn.metrics._pairwise_distances_reduction._middle_term_computer', 'sklearn.metrics._pairwise_distances_reduction._argkmin', 'sklearn.metrics._pairwise_distances_reduction._base', 'sklearn.metrics._pairwise_distances_reduction._radius_neighbors', 'sentence_transformers.SentenceTransformer'], hookspath=[], hooksconfig={}, runtime_hooks=[], excludes=[], win_no_prefer_redirects=False, win_private_assemblies=False, cipher=block_cipher, noarchive=False) pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher) exe = EXE(pyz, a.scripts, a.binaries, a.zipfiles, a.datas, [], name='App', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, upx_exclude=[], runtime_tmpdir=None, console=True, disable_windowed_traceback=False, target_arch=None, codesign_identity=None, entitlements_file=None, icon='Config\\favicon.ico')  I already have "tqdm==4.64.1" installed. I am not sure what is causing this problem. Can someone please help me out on this?

Answer: after adding below instructions in spec file I was able to resolve the issue datas += copy_metadata('tqdm') datas += copy_metadata('regex') datas += copy_metadata('requests') datas += copy_metadata('packaging') datas += copy_metadata('filelock') datas += copy_metadata('numpy') datas += copy_metadata('tokenizers') datas += copy_metadata('importlib_metadata') datas += copy_metadata('tensorflow')
Question: I'm migrating some code from SqlAlchemy 1.3 to SqlAlchemy 1.4 with Postgres 12. I found a query that looks like this: session.query(Horse) .filter(Horse.nicknames.any("charlie", operator=ColumnOperators.ilike))  The type of the column nicknames is Column(ARRAY(String(64))). It seems to me that what this is doing is queryng any Horse whose one of their nicknames is charlie in a case-insensitive (ilike) way. This code seems to work fine in SqlAlchemy==1.3.0 and fails in version 1.4.40 with the following error: sqlalchemy.exc.UnsupportedCompilationError: Compiler <sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7fce54c80f10> can't render element of type <function ColumnOperators.ilike at 0x7fce92944280> (Background on this error at: https://sqlalche.me/e/14/l7de)  What would be an equivalent way of doing this that works, ideally for both versions?

Answer: This query seems to have broken in SQLAlchemy version 1.3.20*. In 1.3.0 it generated this SQL (aliases removed for clarity): SELECT id, nicknames FROM horse WHERE 'charlie' ILIKE ANY (nicknames)  The docs for any mention that it has been superseded by any_, though it doesn't seem to have been formally deprecated. With this knowledge we can build an alternative that generates the same SQL: import sqlalchemy as sa ... session.query(Horse).filter( sa.literal('able').ilike(sa.any_(Horse.nicknames)) )  or in 2.0-style: sa.select(Horse).where( sa.literal('charlie').ilike(sa.any_(Horse.nicknames)) )   * I don't see anything in the release notes regarding this change in behaviour, so it might be a regression, but at this stage it's probably better to go with the any_ construct.
Question: I have a function to retrieve an object in Python using the ctypes module: import ctypes def object_at_addr(obj_addr): try: val = ctypes.cast(obj_addr, ctypes.py_object).value except: return None return val  I know that the id of an object shows the memory address the object is at (for the most part as far as I've seen with builtins and custom objects). On my terminal shell, I tried this on the number 0 and "recursively" found the id of each number to see if it was cyclic in some form. Here is the terminal output: >>> id(0) # num1 4343595216 >>> id(4343595216) # num2 4344636112 >>> id(4344636112) # num3 4344636112  To my surprise, the id of two numbers were the same. However, when I use my function and call object_at_addr(4344636112), it doesn't point to either number, but instead returned a different int value as shown below: >>> object_at_addr(4344636112) 4411205344  How can two different numbers have the same id value? Does the id function actually return the memory address for all objects, or is it only for custom objects, or is it just a Python design choice to make the memory address the id of an object, but it's different for builtin objects?

Answer: The id is always the memory address in the CPython implementation. The reason that you saw numbers with the same id here is that memory addresses can be re-used. The id is only guaranteed to be unique for the lifetime of the object, and since nothing else was holding a reference to the integer 4343595216 it got deleted immediately after the function call returns. That memory location was freed, and then immediately reused by the integer 4344636112 instance. The docs actually mention this explicitly:  Two objects with non-overlapping lifetimes may have the same id() value.
Question: How can we define several tab lengths in a python string? For example, we want to print the keys, value types and values of a dict nicely aligned (with varying sizes of keys and types): my_dict = { "short_key": 4, "very_very_very_very_very_long_keys": 5.0 } formatted_string_1 = '\n'.join([f"{k}:\t({type(v).__name__})\t{v}".expandtabs(10) for k, v in my_dict.items()]) print(f"Option 1 (.expandtabs(10)), first tab is too small:\n{formatted_string_1}") formatted_string_2 = '\n'.join([f"{k}:\t({type(v).__name__})\t{v}".expandtabs(40) for k, v in my_dict.items()]) print(f"\n\nOption 2 (.expandtabs(40)), second tab is too large:\n{formatted_string_2}")  Running this we get: Option 1 (.expandtabs(10)), first tab is too small: short_key: (int) 4 very_very_very_very_very_long_keys: (float) 5.0  and: Option 2 (.expandtabs(40)), second tab is too large: short_key: (int) 4 very_very_very_very_very_long_keys: (float) 5.0  I would like to be able to define a long tab for the first space, and a short tab for the second one, something like .expandtabs([40, 10]), such that we get two nice alignments: short_key: (int) 4 very_very_very_very_very_long_keys: (float) 5.0  Any idea?

Answer: Don't use tabs for alignment. You can specify your desired widths directly in the f-string's format spec: print( '\n'.join( f"{f'{k}:':40}" f"{f'({type(v).__name__})':10}" f"{v:<}" for k, v in my_dict.items() ) )  outputs short_key: (int) 4 very_very_very_very_very_long_keys: (float) 5.0  You can even use variable widths computed from the members of my_dict: key_width = max(len(k) for k in my_dict) + 6 type_width = max(len(type(v).__name__) for v in my_dict.values()) + 5 print( '\n'.join( f"{f'{k}:':{key_width}}" f"{f'({type(v).__name__})':{type_width}}" f"{v:<}" for k, v in my_dict.items() ) )  which again outputs short_key: (int) 4 very_very_very_very_very_long_keys: (float) 5.0
Question: I would like to remove all occurrences of \r from a string as if it was printed via print() and store the result in another variable. Example: >>> s = "hello\rworld" >>> print(s) world  In this example, how do I "print" s to a new variable which then contains the string "world"? Background: I am using the subprocess module to capture the stdout which contains a lot of \r characters. In order to effectively analyze the string I would like to only have the resulting output.

Answer: Using a regex: import re s = "hello\rworld" out = re.sub(r'([^\r]+)\r([^\r\n]+)', lambda m: m.group(2)+m.group(1)[len(m.group(2)):], s)  Output: 'world' More complex example: import re s = "hello\r..\nworld" out = re.sub(r'([^\r]+)\r([^\r\n]+)', lambda m: m.group(2)+m.group(1)[len(m.group(2)):], s)  Output: ..llo world
Question: I'm trying to move both the "$" and totalTransactionCost to the right side of the field. My current code is : print(f"Total Cost Of All Transactions: ${totalTransactionCost:>63,.2f}") The code is able to move the totalTransactionCost to the right side of the field, but how can I include the "$" too?

Answer: You can use nested f-strings, which basically divides formatting into two steps: first, format the number as a comma-separated two-decimal string, and attach the $, and then fill the whole string with leading spaces. >>> totalTransactionCost = 10000 >>> print(f"Total Cost Of All Transactions: {f'${totalTransactionCost:,.2f}':>64}") Total Cost Of All Transactions: $10,000.00
Question: When I try pip install torch, I get ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch Searching on here stackoverflow I find that the issue is I need an older verson of python, currently I'm using 3.11. That post said 3.8 but was written some time ago, so how do I find the latest version of python that will run pytorch? I couldn't find it easily on the PyTorch pages.

Answer: You can always check torch archive or torch nightly to see if your desired version is supported. While the Python3.11 is not officially supported as of now (Feb 11, 2023), if you are on Linux you can install the Python3.11 version of Pytorch 1.13.1: wget https://download.pytorch.org/whl/cu117/torch-1.13.1%2Bcu117-cp311-cp311-linux_x86_64.whl pip3 install torch-1.13.1+cu117-cp311-cp311-linux_x86_64.whl  Just note that this is not yet available on Windows or other major OSes. If you want to give the new version a try on Other OSes such as Windows or Mac, you need to use the nighly builds. For example for Windows inside powershell do : wget https://download.pytorch.org/whl/nightly/cu117/torch-2.0.0.dev20230210%2Bcu117-cp311-cp311-win_amd64.whl -OutFile torch-2.0.0.dev20230210+cu117-cp311-cp311-win_amd64.whl pip install torch-2.0.0.dev20230210+cu117-cp311-cp311-win_amd64.whl
Question: How may I take a shape that was created with more points at its curves and subdivide it so that the points are distributed more equally along the curve? In my research I thought that numpy's interp might be the right function to use, but I don't know what to use for the parameters (x, xp, fp, left, right, & period). Any help would be very appreciated! Here is an animation showing the desired output.  This is the code for the input rounded rectangle: from matplotlib import pyplot as plt import numpy as np x_values = [1321.4, 598.6, 580.6, 563.8, 548.6, 535.4, 524.5, 516.2, 511, 509.2, 509.2, 511, 516.2, 524.5, 535.4, 548.6, 563.8, 580.6, 598.6, 1321.4, 1339.4, 1356.2, 1371.4, 1384.6, 1395.5, 1403.8, 1409, 1410.8, 1410.8, 1409, 1403.8, 1395.5, 1384.6, 1371.4, 1356.2, 1339.4, 1321.4] y_values = [805.4, 805.4, 803.5, 798.3, 790.1, 779.2, 766, 750.8, 734, 716, 364, 346, 329.2, 314, 300.8, 289.9, 281.7, 276.5, 274.6, 274.6, 276.5, 281.7, 289.9, 300.8, 314, 329.2, 346, 364, 716, 734, 750.8, 766, 779.2, 790.1, 798.3, 803.5, 805.4] fig, ax = plt.subplots(1) ax.plot(x_values,y_values) ax.scatter(x_values,y_values) ax.set_aspect('equal') plt.show()  Thank you!

Answer:  from matplotlib import pyplot as plt import numpy as np x = np.array([1321.4, 598.6, 580.6, 563.8, 548.6, 535.4, 524.5, 516.2, 511, 509.2, 509.2, 511, 516.2, 524.5, 535.4, 548.6, 563.8, 580.6, 598.6, 1321.4, 1339.4, 1356.2, 1371.4, 1384.6, 1395.5, 1403.8, 1409, 1410.8, 1410.8, 1409, 1403.8, 1395.5, 1384.6, 1371.4, 1356.2, 1339.4, 1321.4]) y = np.array([805.4, 805.4, 803.5, 798.3, 790.1, 779.2, 766, 750.8, 734, 716, 364, 346, 329.2, 314, 300.8, 289.9, 281.7, 276.5, 274.6, 274.6, 276.5, 281.7, 289.9, 300.8, 314, 329.2, 346, 364, 716, 734, 750.8, 766, 779.2, 790.1, 798.3, 803.5, 805.4]) fig, ax = plt.subplots(1) ax.set_aspect('equal') ax.scatter(x, y, s=40, zorder=3, alpha=0.3) # compute the distances, ds, between points dx, dy = x[+1:]-x[:-1], y[+1:]-y[:-1] ds = np.array((0, *np.sqrt(dx*dx+dy*dy))) # compute the total distance from the 1st point, measured on the curve s = np.cumsum(ds) # interpolate using 200 point xinter = np.interp(np.linspace(0,s[-1], 200), s, x) yinter = np.interp(np.linspace(0,s[-1], 200), s, y) # plot the interpolated points ax.scatter(xinter, yinter, s=5, zorder=4) plt.show()
Question: I am learning about context managers and was trying to build one myself. The following is a dummy context manager that opens a file in read mode (I know I can just do with open(...): .... this is just an example I built to help me understand how to make my own context managers): @contextmanager def open_read(path: str): f = open(path, 'r') print('open') yield f f.close() print('closed') def foo(): try: with open_read('main.py') as f: print(f.readline()) raise Exception('oopsie') except Exception: pass print(f.readline()) foo()  I expect this code to print: open <line 1 of a.txt> closed ValueError: I/O operation on closed file.  But instead it prints: open <line 1 of a.txt> <line 2 of a.txt>  It didn't close the file! This seems to contradict python's docs which state that __exit__ will be called whether the with statement exited successfully or with an exception:  object.exit(self, exc_type, exc_value, traceback) Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.  Interestingly, when I reimplemented the context manager as shown below, it worked as expected: class open_read(ContextDecorator): def __init__(self, path: str): self.path = path self.f = None def __enter__(self): self.f = open(self.path, 'r') print('open') return self.f def __exit__(self, exc_type, exc_val, exc_tb): self.f.close() print('closed')  Why didn't my original implementation work?

Answer: The line f.close() is never reached (we exit that frame early due to unhandled exception), and then the exception was "handled" in the outer frame (i.e. within foo). If you want it to close regardless, you'll have to implement it like that: @contextmanager def open_read(path: str): f = open(path, 'r') try: print('open') yield f finally: f.close() print('closed')  However, I'd like to point out that that the built-in open is already returning a context-manager, and you may be reinventing stdlib contextlib.closing.
Question: I'm not sure where to begin, so looking for some guidance. I'm looking for a way to create some arrays/tables in one process, and have it accessible (read-only) from another. So I create a pyarrow.Table like this: a1 = pa.array(list(range(3))) a2 = pa.array(["foo", "bar", "baz"]) a1 # <pyarrow.lib.Int64Array object at 0x7fd7c4510100> # [ # 0, # 1, # 2 # ] a2 # <pyarrow.lib.StringArray object at 0x7fd7c5d6fa00> # [ # "foo", # "bar", # "baz" # ] tbl = pa.Table.from_arrays([a1, a2], names=["num", "name"]) tbl # pyarrow.Table # num: int64 # name: string # ---- # num: [[0,1,2]] # name: [["foo","bar","baz"]]  Now how do I read this from a different process? I thought I would use multiprocessing.shared_memory.SharedMemory, but that didn't quite work: shm = shared_memory.SharedMemory(name='pa_test', create=True, size=tbl.nbytes) with pa.ipc.new_stream(shm.buf, tbl.schema) as out: for batch in tbl.to_batches(): out.write(batch) # TypeError: Unable to read from object of type: <class 'memoryview'>  Do I need to wrap the shm.buf with something? Even if I get this to work, it seems very fiddly. How would I do this in a robust manner? Do I need something like zmq? I'm not clear how this is zero copy though. When I write the record batches, isn't that serialisation? What am I missing? In my real use case, I also want to talk to Julia, but maybe that should be a separate question when I come to it. PS: I have gone through the docs, it didn't clarify this part for me.

Answer:  Do I need to wrap the shm.buf with something?  Yes, you can use pa.py_buffer() to wrap it: size = calculate_ipc_size(table) shm = shared_memory.SharedMemory(create=True, name=name, size=size) stream = pa.FixedSizeBufferWriter(pa.py_buffer(shm.buf)) with pa.RecordBatchStreamWriter(stream, table.schema) as writer: writer.write_table(table)  Also, for size you need to calculate the size of the IPC output, which may be a bit larger than Table.nbytes. The function you can use for that is: def calculate_ipc_size(table: pa.Table) -> int: sink = pa.MockOutputStream() with pa.ipc.new_stream(sink, table.schema) as writer: writer.write_table(table) return sink.size()   How would I do this in a robust manner?  Not sure of this part yet. In my experience the original process needs to stay alive while the others are reusing the buffers, but there might be a way to get around that. This is likely connected to this bug in CPython: https://bugs.python.org/issue38119  I'm not clear how this is zero copy though. When I write the record batches, isn't that serialisation? What am I missing?  You are correct that writing the Arrow data into an IPC buffer does involve copies. The zero-copy part is when other processes read the data from shared memory. The columns of the Arrow table will reference the relevant segments of the IPC buffer, rather than a copy.
Question: I have created pandas extensions as mentioned here. The extending classes are defined in a module named pd_extensions, and I would like to use them in a different module my_module for example. The two modules are in the same package called source. currently to be able to use the extensions Im importing the pd_extensions module into my_module like this: import source.pd_extensions is there a way to use the extensions I created without importing the module? I find myself importing this module to every module that I want to use the extensions in the package, and I thought there might be a better way of doing it (maybe through the _ _ init _ _ module). I tried just using the extensions without importing the module they are defined in, but it did not work obviously. I'm thinking about importing it in the _ _ init _ _ file so all the modules in the package would have access to it without having to import it themself, but I can't figure out if it's possible.

Answer: I think you can import the extension module in __init__ file since the extension module will first import pandas and then register the accessor therefore the pandas module will be cached in sys.modules and any subsequent import to pandas from other modules will simply retrieve the entry from the cache. here is the simple example: source ├── __init__.py ├── my_module.py └── pd_extension.py  The following are the contente of the files: # pd_extension.py import pandas as pd @pd.api.extensions.register_dataframe_accessor('spam') class Spam: def __init__(self, df): self.df = df @property def shape(self): return self.df.shape   # my_module.py import pandas as pd df = pd.DataFrame([[1, 2, 3], [4, 5, 6]]) print(df.spam.shape)   # __init__.py import source.pd_extension  Now lets test the code by executing my_module.py which works as expected $ python -m source.my_module (2, 3)
Question: My code worked perfectly until yesterday when I updated Google Chrome to version 110.0.5481.77. Now it's not working in headless mode: options.add_argument("--headless")  I even tried adding options.add_argument("--window-size=1280,700") but still not working. Although if I remove the headless option it again works correctly!

Answer: Accroding to this answer and Google Chrome release notes you should add the headless mode option like below: options.add_argument("--headless=new")  and no need to specify the window size