eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: If you readline() from sys.stdin, passing the rest of it to a subprocess does not seem to work. import subprocess import sys header = sys.stdin.buffer.readline() print(header) subprocess.run(['nl'], check=True)  (I'm using sys.stdin.buffer to avoid any encoding issues; this handle returns the raw bytes.) This runs, but I don't get any output from the subprocess; bash$ printf '%s\n' foo bar baz | python demo1.py b'foo\n'  If I take out the readline etc, the subprocess reads standard input and produces the output I expect. bash$ printf '%s\n' foo bar baz | > python -c 'import subprocess; subprocess.run(["nl"], check=True)' 1 foo 2 bar 3 baz  Is Python buffering the rest of stdin when I start reading it, or what's going on here? Running with python -u does not remove the problem (and indeed, the documentation for it only mentions that it changes the behavior for stdout and stderr). But if I pass in a larger amount of data, I do get some of it: bash$ wc -l /etc/services 13921 /etc/services bash$ python demo1.py </etc/services | head -n 3 1 27/tcp # NSW User System FE 2 # Robert Thomas <BThomas@F.BBN.COM> 3 # 28/tcp Unassigned (... traceback from broken pipe elided ...) bash$ fgrep -n 'NSW User System FE' /etc/services 91:nsw-fe 27/udp # NSW User System FE 92:nsw-fe 27/tcp # NSW User System FE bash$ sed -n '1,/NSW User System FE/p' /etc/services | wc 91 449 4082  (So, looks like it eats 4096 bytes from the beginning.) Is there a way I can avoid this behavior, though? I would like to only read one line off from the beginning, and pass the rest to the subprocess. Calling sys.stdin.buffer.readline(-1) repeatedly in a loop does not help. This is actually a follow-up for Read line from shell pipe, pass to exec, and keep to variable but I wanted to focus on this, to me, surprising aspect of the problem in that question.

Answer: This is because sys.stdin is created using the built-in open function in the default buffered mode, which uses a buffer of size io.DEFAULT_BUFFER_SIZE, which on most systems is either 4096 or 8192 bytes. To make the parent process consume precisely one line of text from the standard input, you can therefore open it with the buffer disabled by passing 0 as the buffering argument to the open or os.fdopen function: # subp1.py import os import sys import subprocess # or with the platform-dependent device file: # unbuffered_stdin = open('/dev/stdin', 'rb', buffering=0) unbuffered_stdin = os.fdopen(sys.stdin.fileno(), 'rb', buffering=0) print(unbuffered_stdin.readline()) subprocess.run(['nl'], check=True)  so that: printf "foo\nbar\n" | python subp1.py  would then output: b'foo\n' 1 bar
Question: I have a 3D circular paraboloid surface and I would like to plot a spiral that starts from an arbitrary point on the surface and goes down while "hugging" the surface. This is my attempt so far: import numpy as np import matplotlib.pyplot as plt fig = plt.figure() ax = plt.axes(projection='3d') # Surface ------------------ # Create the mesh in polar coordinates and compute corresponding Z r0 = 5 r = np.linspace(0, r0, 50) p = np.linspace(0, 2*np.pi, 50) R, P = np.meshgrid(r, p) Z = -R**2 + r0**2 # Express the mesh in the cartesian system X, Y = R*np.cos(P), R*np.sin(P) # Plot the surface ax.plot_surface(X, Y, Z, linewidth=0, antialiased=False, alpha=0.2) # Spiral ------------------- u = np.arange(0, 29, 0.1) x = 0.17*u*np.cos(u) y = 0.17*u*np.sin(u) z = -0.15*u/np.pi*(x**2 + y**2) + r0**2 # Plot spiral ax.plot3D(x, y, z, 'r') plt.show()   However, my spiral is not actually following the surface. I also tried this: x = [] y = [] z = [] for i in range(50): x.append(X[i,i]) y.append(Y[i,i]) z.append(-(X[i,i]**2 + Y[i,i]**2) + r0**2) ax.plot3D(x, y, z, 'b')   which is going around the surface but I don't know how to make it do more circles around the surface. Any ideas?

Answer: The formula in the second attempt is correct. I get what you want if I use the same formula in your first attempt. The line z = -0.15*u/np.pi*(x**2 + y**2) + r0**2 needs to be replaced with -(x**2 + y**2) + r0**2. For reproducibility: %matplotlib notebook import numpy as np import matplotlib.pyplot as plt fig = plt.figure() ax = plt.axes(projection='3d') # Surface ------------------ # Create the mesh in polar coordinates and compute corresponding Z r0 = 5 r = np.linspace(0, r0, 50) p = np.linspace(0, 2*np.pi, 50) R, P = np.meshgrid(r, p) Z = -R**2 + r0**2 # Express the mesh in the cartesian system X, Y = R*np.cos(P), R*np.sin(P) # Plot the surface ax.plot_surface(X, Y, Z, linewidth=0, antialiased=False, alpha=0.2) # Spiral ------------------- # Attempt 1 u = np.arange(0, 29, 0.1) x = 0.17*u*np.cos(u) y = 0.17*u*np.sin(u) z = -(x**2 + y**2) + r0**2 # z = -0.15*u/np.pi*(x**2 + y**2) + r0**2 # Plot spiral ax.plot3D(x, y, z, 'r') plt.show()  Output is as shown below:
Question: Suppose we wish to extract the minimum value from a dictionary like so scores = { 0:1.3399288498085087, 1:1.2672683347433629, 3:1.6999159970296505, 4:1.8410942584597279, 5:1.336658057628646 } #find minimum value in dictionary minimum_value = min(scores.values()) #get keys with minimal value using list comprehension minimum_keys = [key for key in scores if scores[key]==minimum_value] minimum_keys  This returns the key with the lowest value. However, what if I wish to extract the minimum 2 keys and put them into a list? What if I wanted the minimum 20? How would I go about doing this for an arbitrary number of minimal values desired?

Answer: In fact, the problem is simpler than you think: scores = { 0:1.3399288498085087, 1:1.2672683347433629, 3:1.6999159970296505, 4:1.8410942584597279, 5:1.336658057628646 } # minimum key print(min(scores, key=scores.get)) # n minimum keys print(sorted(scores, key=scores.get)[:3])  Output: 1 [1, 5, 0]  Both min and sorted allow you to provide a key which is some function to be called with a value, to compute an associated value to be used for sorting. By providing scores.get as that function, you can sort keys by their matching value, which is what you want.
Question: Say I want to share a dictionary between processes. If I have defined a manager, what is the difference between instantiating a pool using manager.Pool() and multiprocessing.Pool()? Ex: What is the difference between the two with statements in main_1 and main_2? import multiprocessing as mp import time from random import random def func(d): pid = mp.current_process().pid time.sleep(3 * random()) d[pid] = 'I added this' def main_1(): print('Start of main 1') with mp.Manager() as manager: d = manager.dict() with manager.Pool() as pool: # using manager pool.map(func, [d] * 4) print(d) def main_2(): print('Start of main 2') manager = mp.Manager() try: d = manager.dict() with mp.Pool() as pool: # using multiprocessing pool.map(func, [d] * 4) print(d) finally: manager.shutdown() if __name__ == '__main__': main_1() main_2()  More generally Are all processes started after a Manager exists in the scope automatically served by it?

Answer: Both manager.Pool and multiprocessing.Pool in Python are used for creating a pool of worker processes to execute tasks in parallel. However, there are some differences between them. multiprocessing.Pool is a built-in class provided by the Python multiprocessing module. It creates a pool of worker processes, where each process executes tasks independently. The tasks can be submitted to the pool as functions or methods to be executed in parallel. On the other hand, manager.Pool is a class provided by the Python multiprocessing.managers module. It also creates a pool of worker processes like multiprocessing.Pool. However, manager.Pool is used for creating a pool of processes that can be shared among multiple Python processes. This means that the processes in the pool can be accessed and used by multiple Python processes, making it a good choice for distributed computing.
Question: How to structurally pattern match builtin type slice in python? Somewhy the following code does not work: def __getitem__(self, index): match index: case int(i): ... case slice(start, stop, step): ... case _: ...  and I completely do not understand why.

Answer: Try: class Example: def __getitem__(self, index): match index: case int(i): print('int', i) case slice(start=start, stop=stop, step=step): print('slice', start, stop, step) case _: print('default') e = Example() e[1] e[1:2]  Prints: int 1 slice 1 2 None
Question: code: df['Rep'] = df['Rep'].str.replace('\\n', ' ')  issue: if the df['Rep'] is empty or null ,there will be an error: Failed: Can only use .str accessor with string values!  is there anyway can handle the situation when the column value is empty or null? If it is empty or null ,just ignore that row

Answer: By default the empty series dtype will be float64. You can do a workaround using the astype: df['Rep'] = df['Rep'].astype('str').str.replace('\\n', ' ')  Test code: df = pd.DataFrame({'Rep': []}) # works df['Rep'] = df['Rep'].astype('str').str.replace('\\n', ' ') # doesn't work df['Rep'] = df['Rep'].str.replace('\\n', ' ')  I don't know which version of pandas you are using by they will changed to object the default dtype. Edit: Still won't work with object, just tested using the latest version of pandas (2.0.0). Source
Question: i just took over a project in python after a year and i wanted to rebuild it with customtkinter using the documentation. Here is the code: import customtkinter import tkinter from pytube import YouTube from PIL import Image customtkinter.set_appearance_mode("system") customtkinter.set_default_color_theme("blue") app = customtkinter.CTk() app.geometry("800x760") app.title("YouTube Unlocked") logo = customtkinter.CTkImage(dark_image=Image.open("G:\\ytdownloader\\assets\\icon.png"), size=(260,190)) logoDisplay = customtkinter.CTkLabel(app, text="", image=logo) logoDisplay.pack(padx=10, pady=10) linkLabel = customtkinter.CTkLabel(app, text="Video link", font=("Arial", 15)) linkLabel.pack(padx=10, pady=10) linkTextbox = customtkinter.CTkEntry(app, width=400) linkTextbox.pack(padx=10, pady=10) radio_var = tkinter.IntVar(0) radiobutton_1 = customtkinter.CTkRadioButton(app, text="Video", variable=radio_var, value=1) radiobutton_2 = customtkinter.CTkRadioButton(app, text="Audio", variable=radio_var, value=2) radiobutton_1.pack(padx=10, pady=10) radiobutton_2.pack(padx=10, pady=10) app.mainloop()  But after writing the radio buttons code (always from the documentation) I got this error: Traceback (most recent call last): File "g:\ytdownloader\main.py", line 23, in <module> radio_var = tkinter.IntVar(0) File "C:\Users\gp_ga\AppData\Local\Programs\Python\Python310\lib\tkinter\__init__.py", line 564, in __init__ Variable.__init__(self, master, value, name) File "C:\Users\gp_ga\AppData\Local\Programs\Python\Python310\lib\tkinter\__init__.py", line 372, in __init__ self._root = master._root() AttributeError: 'int' object has no attribute '_root'  Is there any way to fix this? Thanks! (by the way if it's a bad english I want to let you know that i'm using google translate)

Answer: tkinter.IntVar() already has a default value of 0. The documentation says the following about the arguments that tkinter.IntVar() can take:  tkinter.IntVar(master=None, value=None, name=None)  Since master comes first and you provided 0 as the first positional argument, it believes that it is the master argument. What you want to do is specify the argument value. Like so: radio_var = tkinter.IntVar(value=0)
Question: So, I have to run a recursive function times but it's going to take too long to actually print out all the values. Thus, my professor recommended using Excel but I don't know Excel at all. I need help converting the code into Excel. It's probably easy for someone who knows Excel. def a(n): k=3.8 if n==0: return .5 else: return k*a(n-1)*(1-a(n-1)) for i in range(100): print(a(i))  All i know is that you use the lambda() function in excel

Answer: You don't need to use excel. You just need to use a better algorithm. The easiest way to prevent the exponential time complexity is don't re-calculate the same value twice: def a(n): k = 3.8 if n==0: return .5 else: x = a(n - 1) return k*x*(1-x) for i in range(100): print(a(i))  In Python, you should avoid recursion, though, since Python doesn't optimize recursion and you will run out of stack space. This is easy to convert to an iterative algorithm, though: def b(n): k = 3.8 prev = curr = 0.5 for i in range(1, n + 1): curr = k * prev * (1 - prev) prev = curr return curr
Question: I am trying calculate RSI indicator. For that I need rolling-mean gain and loss. I would like to calculate rolling mean ignoring null values. So mean would be calculated by sum and count on existing values. Example: window_size = 5 df = DataFrame(price_change: { 1, 2, 3, -2, 4 }) df_gain = .select( pl.when(pl.col('price_change') > 0.0) .then(pl.col('price_change')) .otherwise(None) .alias('gain') ) # UNKOWN HOW TO GET WANTED RESULT: rol_mean_gain = df_gain.select( pl.col('gain').rolling_mean(window_size=window_size, ignore_null=True) )  So that rol_mean_gain be calculated: [1, 2, 3, skip, 4] / 4 (not 5) I know Pandas has .mean(skipna=True) or .apply(pandas.np.nanmean) But as far as I am aware polars does not provide such API.

Answer: I think the skipping of nulls is implied, see the min_periods description in the docs for this method. df_gain.select(pl.col('gain').rolling_mean(window_size=window_size, min_periods=1))  Gives me a column of 1.0, 1.5, 2.0, 2.0, 2.5. Note how the last two columns skips the null correctly.
Question: I have Polars dataframe with a bunch of columns I need to find the column with, for example, the largest sum. The below snippet sums all of the columns: df = pl.DataFrame( { "a": [0, 1, 3, 4], "b": [0, 0, 0, 0], "c": [1, 0, 1, 0], } ) max_col = df.select(pl.col(df.columns).sum())  shape: (1, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 8 ┆ 0 ┆ 2 │ └─────┴─────┴─────┘  But I'm missing the last step of selecting the column with the largest value?

Answer: I would do this as a unpivot/filter.  df \ .select(pl.all().sum()) \ .unpivot() \ .filter(pl.col('value')==pl.col('value').max())  If you want the original shape then a single chain is a bit tougher. I'd just do it like this instead. allcalc=df \ .select(pl.all().sum()) allcalc.select(allcalc.unpivot().filter(pl.col('value')==pl.col('value').max()) \ .get_column('variable').to_list())  The above works if there is a tie, for instance if you have: df=pl.DataFrame( { "a": [0, 1, 3, 4], "b": [0, 0, 0, 0], "c": [1, 0, 1, 6], } )  then you'll get 'a' and 'c' in either case.
Question: I am working with the Python Polars library for data manipulation on a DataFrame, and I am trying to change the position of a single column. I would like to move a specific column to a different index while keeping the other columns in their respective positions. One way of doing that is using select, but that requires giving a complete order for all the columns which I don't want to do. import polars as pl # Create a simple DataFrame data = { 'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9], 'D': [10, 11, 12] } df = pl.DataFrame(data)  I want to move column 'C' to index 1, so the desired output should be: shape: (3, 4) ┌─────┬─────┬─────┬──────┐ │ A │ C │ B │ D │ │ --- │ --- │ --- │ ---- │ │ i64 │ i64 │ i64 │ i64 │ ╞═════╪═════╪═════╪══════╡ │ 1 │ 7 │ 4 │ 10 │ ├─────┼─────┼─────┼──────┤ │ 2 │ 8 │ 5 │ 11 │ ├─────┼─────┼─────┼──────┤ │ 3 │ 9 │ 6 │ 12 │ └─────┴─────┴─────┴──────┘ 

Answer: Some attempts: df.drop("C").insert_column(1, df.get_column("C"))  df.select(df.columns[0], "C", pl.exclude(df.columns[0], "C"))  cols = df.columns cols[1], cols[2] = cols[2], cols[1] # cols[1:3] = cols[2:0:-1] df.select(cols)  shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ C ┆ B ┆ D │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 1 ┆ 7 ┆ 4 ┆ 10 │ │ 2 ┆ 8 ┆ 5 ┆ 11 │ │ 3 ┆ 9 ┆ 6 ┆ 12 │ └─────┴─────┴─────┴─────┘
Question: I have following problem. Let's say I have two dataframes df1 = pl.DataFrame({'a': range(10)}) df2 = pl.DataFrame({'b': [[1, 3], [5,6], [8, 9]], 'tags': ['aa', 'bb', 'cc']}) print(df1) print(df2)  shape: (10, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 0 │ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ 5 │ │ 6 │ │ 7 │ │ 8 │ │ 9 │ └─────┘ shape: (3, 2) ┌───────────┬──────┐ │ b ┆ tags │ │ --- ┆ --- │ │ list[i64] ┆ str │ ╞═══════════╪══════╡ │ [1, 3] ┆ aa │ │ [5, 6] ┆ bb │ │ [8, 9] ┆ cc │ └───────────┴──────┘  I need to mark/tag rows in dataframe df1 based on values of dataframe df2, so I can get following dataframe print(pl.DataFrame({'a': range(10), 'tag': ['NA', 'aa', 'aa', 'aa', 'NA', 'bb', 'bb', 'NA', 'cc', 'cc']}))  shape: (10, 2) ┌─────┬─────┐ │ a ┆ tag │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪═════╡ │ 0 ┆ NA │ │ 1 ┆ aa │ │ 2 ┆ aa │ │ 3 ┆ aa │ │ 4 ┆ NA │ │ 5 ┆ bb │ │ 6 ┆ bb │ │ 7 ┆ NA │ │ 8 ┆ cc │ │ 9 ┆ cc │ └─────┴─────┘  So list in column b of df2 indicates start and end values for column a of df1 that needs to be tagged with what's in column tags. Thanks

Answer: You could create the ranges and "flatten" the frame:  .int_ranges() .explode()  (df2 .with_columns(pl.int_ranges(pl.col("b").list.first(), pl.col("b").list.last() + 1)) .explode("b") )  shape: (7, 2) ┌─────┬──────┐ │ b ┆ tags │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪══════╡ │ 1 ┆ aa │ │ 2 ┆ aa │ │ 3 ┆ aa │ │ 5 ┆ bb │ │ 6 ┆ bb │ │ 8 ┆ cc │ │ 9 ┆ cc │ └─────┴──────┘  Matching could then be done with a LEFT JOIN df1.join( df2.with_columns( pl.int_ranges(pl.col("b").list.first(), pl.col("b").list.last() + 1)).explode("b"), left_on = "a", right_on = "b", how = "left" )  shape: (10, 2) ┌─────┬──────┐ │ a ┆ tags │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪══════╡ │ 0 ┆ null │ │ 1 ┆ aa │ │ 2 ┆ aa │ │ 3 ┆ aa │ │ 4 ┆ null │ │ 5 ┆ bb │ │ 6 ┆ bb │ │ 7 ┆ null │ │ 8 ┆ cc │ │ 9 ┆ cc │ └─────┴──────┘
Question: I have an Ibis table named t. Its column names are all lowercase. I want to change them all to uppercase. How can I do that?

Answer: The rename method of Ibis table objects renames columns. It can be used to make all the column names uppercase like this: t = t.rename(dict(zip([x.upper() for x in t.columns], t.columns)))  rename also provides a shortcut for this: t = t.rename("ALL_CAPS")  The above works in Ibis version 7.0.0 or newer. In older versions of Ibis, you can use relabel: t = t.relabel(dict(zip(t.columns, [x.upper() for x in t.columns])))
Question: I really like the factory boy style of generated factories that can handle things like sequences, complex relationships etc. For a FastAPI app with fully async database access using factory boy seems likely problematic. There is dated discussion here and an old PR to add async support that seems stuck. Is there a good solution for these kinds of fixtures that has full async support?

Answer: I haven't seen further progress from factory boy on this issue, but ultimately implemented a solution using pytest fixtures as factories that is working well for me. The core idea is to build fixtures that return a factory method that can be used in tests. Here is a concrete example that generates users: @pytest.fixture(scope="function") def user_factory(db: Session): """A factory for creating users""" last_user = 0 def create_user( email=None, name=None, role=None, org: Organization | None = None ) -> User: """Return a new user, optionally with role for an existing organization""" nonlocal last_user last_user += 1 email = email or f"user{last_user}@example.com" name = name or f"User {last_user}" user = User(email=email, name=name, auth_id=auth_id) db.add(user) if role: role = OrganizationRole(user_id=user_id, organization_id=org.id, role=role) db.add(role) db.commit() db.refresh(user) return user return create_user # use in test def test_something(user_factory) -> None: user = user_factory() # ...  I picked an example with SQLAlchemy ORM for simplicity of demonstrating the concept but you can persist however you like, pull in other factories as dependencies, etc. There is a pretty good discussion of how to approach this in this article about using factories as fixtures. This article also has good ideas for how to address sequences, constraints and a lot of the things I would use factory boy for by just using simple python.
Question: I often need to retrieve a row from a Polars DataFrame given a collection of column values, like I might use a composite key in a database. This is possible in Polars using DataFrame.row, but the resulting expression is very verbose: row_index = {'treatment': 'red', 'batch': 'C', 'unit': 76} row = df.row(by_predicate=( (pl.col('treatment') == row_index['treatment']) & (pl.col('batch') == row_index['batch']) & (pl.col('unit') == row_index['unit']) ))  The most succinct method I've found is from functools import reduce from operator import and_ expr = reduce(and_, (pl.col(k) == v for k, v in row_index.items())) row = df.row(by_predicate=expr)  But that is still verbose and hard to read. Is there an easier way? Possibly a built-in Polars functionality I'm missing?

Answer: (a == b) & (c == d) will return true if all of the conditions are true. Another way to express this is with pl.all_horizontal() pl.all_horizontal(a == b, c == d)   pl.any_horizontal() can be used for "logical OR"  To which you can pass your comprehension directly: expr = pl.all_horizontal( pl.col(k) == v for k, v in row_index.items() ) df.row(by_predicate=expr)
Question: In light of some distributions (Debian at least) steering away from python3 -m pip install numpy what should I do with my user packages that I had installed with python3 -m pip install --user numpy (for instance)? They are located in python3 -m site --user-site : ~/.local/lib/python3.11/site-packages/. I don't want to:  delete /usr/lib/python3.x/EXTERNALLY-MANAGED install pipx sometimes, of course, cannot find an apt managed package  I'm all for using venv, but do I make some ~/.local/lib/python3/packages folder and include it in a PYTHON_PATH, but also make that folder a venv virtual environment? Should I move what is in my ~/.local/lib/python3.11/site-packages/ folder to this new folder, or re-download? For someone who uses Python and 'standard' packages, I really don't need to worry about mixing things too much, but to me it seems (only for me as a low-key user) that functionally there is little difference between venv installs in one directory (one for each project) and --user installs in another. I've looked around a lot about this but couldn't find a "here's what to do" tutorial on adopting the change. Or maybe I am missing some big picture... How do I solve "error: externally-managed-environment" everytime I use pip3? pip install -r requirements.txt is failing: "This environment is externally managed" pip install -r requirements.txt is failing: "This environment is externally managed"

Answer: I decided to use pipenv to manage my packages, but to install it, I have to do a pip install. I wanted to do this just for my user, so I created a local venv for it, mkdir -p ~/.local/share/ apt install python3-pip python3 -m venv ~/.local/share/pipenv source ~/.local/share/pipenv/bin/activate  and installed pipenv on it: pip install pipenv  To now use pipenv with my user, I added the path to the pipenv bin venv to my PATH: export PATH=$HOME/.local/share/pipenv/bin:$PATH  Now I can use pipenv to manage the project venvs. It's kinda convoluted, I know, but Python can be quite convoluted with packages.