eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Does polars.DataFrame.partition_by preserves the order of rows within each group? I understand that group_by does, even when maintain_order=False. From documentation: Within each group, the order of rows is always preserved, regardless of this argument. But nothing is mentioned for the partition_by operation. I guess this means the order is not guaranteed to be preserved, but looking for confirmation, since from a few tests I did the resulting dataframes (partitioned) always respected the original order. Here is a the code I used for some toy experiment: df = pl.DataFrame({ "a" : np.arange(100000000), "b": np.random.randint(0,50,100000000) } ) all_dfs = df.partition_by("b", as_dict=True) for key, df in all_dfs.items(): assert df["a"].is_sorted() 

Answer: partition_by is implemented by just doing a group_by and extracting the groups into separate DataFrames. I see no reason why we would change that, so I think it's safe to assume the order within each group is preserved, at least with the default arguments. I'll see if we can get the docs to match group_by's docs in that regard.
Question: I have a table that is laid out somewhat like this:    t linenum many other columns     1234567 0 ...   1234568 0 ...   1234569 0 ...   1234570 1 ...   1234571 1 ...    Except it is very, very large. As in, the raw .dat files can get up to 20 gb. I have them converted into .h5 files so they are slightly smaller, but still large (about half the size, I'd say.) I want to add a column that is time within line, so it subtracts the first time value for the line from each time, so I end up having something like this:    t linenum time within line     1234567 0 0   1234568 0 1   1234569 0 2   1234570 1 0   1234571 1 1    The thing is, while I know that doing an operation on the whole dataframe at once is much faster, I haven't been able to figure out how to do this without using a for loop, since the number that needs to be subtracted depends on linenum, and it takes ages. (Yesterday, I tested this on a file about 9gb big, and I gave up and went home after it had been processing for half an hour, only to find this morning that my computer had restarted overnight so the jupyter server had to restart and I lost the processed dataframe...) Here is the relevant parts of the code I currently have: import pandas as pd file = [h5 file address] df = pd.read_hdf(file) for linenum in pd.unique(df['linenum']): line_df = df.loc[df['linenum'] == linenum] first_t = int(line_df['t'].iloc[0]) df.loc[df['linenum'] == linenum, 't_adjusted'] = (df.loc[df['linenum'] == linenum, 't'] - first_t)  Is there any way to do this without a for loop, and if not, is there any way to make it faster? I'm trying to graph one of the other columns using matplotlib.pyplot.tricontourf, with linenum on the x axis and time within line on the y axis, if that's relevant at all. There is another column I can use as a workaround because it's approximately proportional to time within line but I'd prefer to find a way to use the time. Thank you! Edit: Also, if it's relevant, I am using Python 3.7. For some reason some of the computers my programs have to run on at my work are still on Windows 7 so I can't update...

Answer: You can use a groupby on 'linenum' and then transform to populate each group df['timewithinline'] = df.groupby('linenum')['t'].transform(lambda x: x - min(x))  If the times are already sorted, you can use: df['timewithinline'] = df.groupby('linenum')['t'].transform(lambda x: x - x.iloc[0])
Question: I have a pandas dataframe ('df3') which columns I would like to rename.It is a subset of another dataframe, and I'm working in a jupyter notebook. Getting all infos from the dataframe structure: df3.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 29 entries, 104 to 132 Data columns (total 15 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Zählung in 28 non-null object 1 Unnamed: 17 29 non-null object 2 Unnamed: 18 29 non-null object 3 Unnamed: 19 29 non-null object 4 Unnamed: 20 29 non-null object 5 Unnamed: 21 29 non-null object 6 Unnamed: 40 29 non-null object 7 Unnamed: 41 29 non-null object 8 Unnamed: 42 29 non-null object 9 Unnamed: 43 28 non-null object 10 Unnamed: 44 28 non-null object 11 Unnamed: 63 29 non-null object 12 Unnamed: 64 29 non-null object 13 Unnamed: 65 29 non-null object 14 Unnamed: 66 28 non-null object dtypes: object(15) memory usage: 3.5+ KB  The columns need to be renamed by their index, so I create a dict with the index as keys and the new names as values: col_names= {0:'Zeit', 1:'ZU_PKW', 2: 'ZU_LKW1', 3: 'ZU_LKW2', 4: 'ZU_SV', 5: 'ZU_SV%', 6:'AB_PKW', 7: 'AB_LKW1', 8: 'AB_LKW2', 9: 'AB_SV', 10: 'AB_SV%', 11:'ALL_PKW', 12: 'ALL_LKW1', 13: 'ALL_LKW2', 14: 'ALL_SV' }  I tried to rename the columns with all options using the 'rename'-function: 1. Create a new dataframe with renamed columns without using inplace=True df4 = df3.rename(index=col_names)  Result: no error, but the columns are not renamed 2. Create a new dataframe with renamed columns by using inplace=True df4 = df3.rename(index={col_names, inplace = True)  Result: Error: A value is trying to be set on a copy of a slice from a DataFrame And the resulting df is None. 3. Change the dataframe directly by using inplace=True  df3.rename(index={col_names, inplace = True)  Result: `Error: A value is trying to be set on a copy of a slice from a DataFrame and the df3 stays unchanged. 4. Specifying also axis=1 df3.rename(index={col_names, inplace = True)  Returns TypeError: Cannot specify both 'axis' and any of 'index' or 'columns' I feel I'm running out of options. What am I doing wrong? Could it have something to do with working in the jupyter notebook environment?

Answer: If need rename by index of columns set values in list comprehension: df.columns = [col_names.get(i, i) for i in range(len(df.columns))]  Or: df.columns = pd.Series(range(len(df.columns))).replace(col_names)  If need set first N values by values of dictionary: df.columns = list(col_names.values())[:len(df.columns)]  But if really need rename need change columns names to RangeIndex first: df = df.set_axis(range(len(df.columns)), axis=1).rename(columns=col_names)
Question: I have a complete "working" python code which is supposed to contain two threads which run simultaneously, which populate some lists in a dict, and when the user presses CRTL-C these two threads should be stopped and some output from both threads should be written to a file: import sys import time import threading import signal from functools import partial messages = {} lock = threading.Lock() class Handler: def __init__(self, port): self.port = port def run(self): while True: time.sleep(1) with lock: messages[self.port].append(time.time()) def signal_handler(filename, sig, frame): with lock: with open(filename, "w") as fileout: json.dump(messages, fileout) sys.exit(0) output = "test.out" signal.signal(signal.SIGINT, partial(signal_handler, output)) for port in [1,2]: messages[port] = [] handler = Handler(port) print("debug1") t = threading.Thread(target=handler.run()) print("debug2") t.daemon = True t.start() threads.append(t) # Keep the main thread running, waiting for CTRL-C try: while True: pass except KeyboardInterrupt: signal_handler(output, signal.SIGINT, None)  However, this code blocks execution after the first debug1 has been printed. How to "unblock" this line so the two threads are started until the user presses CRTL-C (and the output is saved to a file)? ...The above code is just a template of a more complicated code that actually does something useful...

Answer: Apart from various typos (forgotten import json and threads=[]) the main problem is in t = threading.Thread(target=handler.run()). The target parameter is expected to be a function object, not the result of the call of that function. Here this code immediately calls the never ending function to assign its never coming result to t, actually blocking the main program. The solution is just to remove the parens: t = threading.Thread(target=handler.run)
Question: I have the following Polars.DataFrame: df = pl.DataFrame( { "timestamp": [1, 2, 3, 1, 2, 3], "var1": [1, 2, 3, 3, 4, 5], "group": ["a", "a", "a", "b", "b", "b"], } ) print(df) out: shape: (6, 3) ┌───────────┬──────┬───────┐ │ timestamp ┆ var1 ┆ group │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════════╪══════╪═══════╡ │ 1 ┆ 1 ┆ a │ │ 2 ┆ 2 ┆ a │ │ 3 ┆ 3 ┆ a │ │ 1 ┆ 3 ┆ b │ │ 2 ┆ 4 ┆ b │ │ 3 ┆ 5 ┆ b │ └───────────┴──────┴───────┘  I want to replace the values of group b with the values of group a that are having the same timestamps. Desired output: shape: (6, 3) ┌───────────┬──────┬───────┐ │ timestamp ┆ var1 ┆ group │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════════╪══════╪═══════╡ │ 1 ┆ 1 ┆ a │ │ 2 ┆ 2 ┆ a │ │ 3 ┆ 3 ┆ a │ │ 1 ┆ 1 ┆ b │ │ 2 ┆ 2 ┆ b │ │ 3 ┆ 3 ┆ b │ └───────────┴──────┴───────┘  I have the current solution with generating a helper df: def group_value_replacer( df: pl.DataFrame, target_group_col: str, target_var: str, target_group: str, ): helper_df = df.filter(pl.col(target_group_col) == target_group) df = df.drop(target_var).join( helper_df.drop(target_group_col), on=["timestamp"], how="left", ) return df group_value_replacer(df, "group", "var1", "a") out： shape: (6, 3) ┌───────────┬───────┬──────┐ │ timestamp ┆ group ┆ var1 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 │ ╞═══════════╪═══════╪══════╡ │ 1 ┆ a ┆ 1 │ │ 2 ┆ a ┆ 2 │ │ 3 ┆ a ┆ 3 │ │ 1 ┆ b ┆ 1 │ │ 2 ┆ b ┆ 2 │ │ 3 ┆ b ┆ 3 │ └───────────┴───────┴──────┘  I want to improve the solution by using Polars.Expr: For example, is there a way for me to achieve the same operation using expressions like df.with_columns(pl.col(target_var).operationxx).

Answer: I think for generic solution your approach with join works fine, you could probably try something like this as well:  filter() to filter var1 column to leave only values where group == a first() to get the value. over() to limit it to certain timestamp. coalesce() to fallback to actual value if value for group == a doesn't exist.  df.with_columns( pl.coalesce( pl.col.var1.filter(pl.col.group == "a").first().over("timestamp"), pl.col.var1 ) )  ┌───────────┬──────┬───────┐ │ timestamp ┆ var1 ┆ group │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════════╪══════╪═══════╡ │ 1 ┆ 1 ┆ a │ │ 2 ┆ 2 ┆ a │ │ 3 ┆ 3 ┆ a │ │ 1 ┆ 1 ┆ b │ │ 2 ┆ 2 ┆ b │ │ 3 ┆ 3 ┆ b │ └───────────┴──────┴───────┘  You can also skip coalesce() if you don't need a fallback. df.with_columns( pl.col.var1.filter(pl.col.group == "a").first().over("timestamp") )
Question: JoinStrategy is an input to join: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join.html My static type checking tool seems to be able to get a hold of JoinStrategy, but I'm not sure how/from where.  Usually, type stub packages are available on PyPI, but nothing obvious stands out in this case: https://pypi.org/user/ritchie46/ How do I import JoinStrategy (or other type definitions provided by Polars) for my own use?

Answer: Types can be provided by one of two things: a separate type package, and by the same package. In this case it's being provided by the same package. If you read the source code, you can find where the type comes from. Example: from polars._typing import JoinStrategy  Note that _typing denotes that this is part of the Polars private API, and is subject to change between releases. You can also print out the value of JoinStrategy: >>> JoinStrategy typing.Literal['inner', 'left', 'right', 'full', 'semi', 'anti', 'cross', 'outer']  You can also use this as a type definition. import typing JoinStrategy = typing.Literal['inner', 'left', 'right', 'full', 'semi', 'anti', 'cross', 'outer']  This has the advantage of not using the private API, but the disadvantage that if Polars adds a new JoinStrategy, your code won't automatically allow that in this type.
Question: I am trying to generate a wheel file using bazel, for a target that has pybind dependencies. The package by itself works fine (though testing), but when I'm packing it, the .so file is missing from the site_packges folder. This is my build file: load("@pybind11_bazel//:build_defs.bzl", "pybind_extension") load("@python_pip_deps//:requirements.bzl", "requirement") load("@rules_python//python:defs.bzl", "py_library", "py_test") load("@rules_python//python:packaging.bzl", "py_wheel", "py_package") # wrapper for so file py_library( name = "example", srcs = ["example.py","__init__.py"], deps = [ ], data = [":pyexample_inf"], imports = ["."], ) # compile pybind cpp pybind_extension( name = "pyexample_inf", srcs = ["pyexample_inf.cpp"], deps = [], linkstatic = True, ) # test wrapper py_test( name = "pyexample_test", srcs = ["tests/pyexample_test.py"], deps = [ ":example", ], ) # Use py_package to collect all transitive dependencies of a target, # selecting just the files within a specific python package. py_package( name = "pyexample_pkg", visibility = ["//visibility:private"], # Only include these Python packages. deps = [":example"], ) # using pip, this copies the files to the site_packges, but not the so file py_wheel( name = "wheel", abi = "cp311", author = "me", distribution = "example", license = "Apache 2.0", platform = select({ "@bazel_tools//src/conditions:linux_x86_64": "linux_x86_64", }), python_requires = ">=3.9.0", python_tag = "cpython", version = "0.0.1", deps = [":example"], )  How can I make the py_wheel copy the so file?

Answer: try deps = [":example.so"]
Question: My configuration is very basic. A simple supabase database with one table. I use supabase-py to interact with it. The problem is that I always get an empty list : from supabase import create_client URL = "MY_URL_HERE" API_KEY = "MY_API_KEY_HERE" supabase = create_client(URL, API_KEY) response = supabase.table("prod_vxf").select("*").execute() print(respnse.data) # []  After checking some similar topics like this one, it seems that the only solution is by turning off RLS. So I went to the dashboard and turned off the RLS for the table prod_vxf and it worked. Now, the code above gives a non empty list : print(response.data) [ {"id": 1, "created_at": "2024-01-01T00:00:00+00:00"}, {"id": 2, "created_at": "2024-01-02T00:00:00+00:00"}, {"id": 3, "created_at": "2024-01-03T00:00:00+00:00"}, ]  But what is very confusing is the warning below that hits my screen when I try to turn off the RLS for a given table in supabase dashboard. Does it mean that anyone on the internet (even without knowing url + api key) can access (read and write) my database and its tables ? Honestly, I'm super confused by the term publicly used by the warning. 

Answer:  Does it mean that anyone on the internet (even without knowing url + api key) can access (read and write) my database and its tables ?  It's publicly as in the database role PUBLIC:  The special “role” name PUBLIC can be used to grant a privilege to every role on the system.  That is every role inside that db, not the general, public internet public. It's not like that change suddenly peels away all security from your entire database cluster. The answer to your question is a hard no. Everyone still needs to know the connection string to get in or the URL plus an API key to perform operations indirectly. Anyone that does get in either way, still needs to do so as a user that has the privileges to access the table. Only then can they CRUD records in that table freely, but since RLS is a per-table setting, it only applies to that one single table. You can check others to confirm they are still protected. The Auth policies they also mention won't work because they rely on RLS. If it's just you and your friends working on the app, this doesn't matter much. If you plan to make the app available to a broader audience, make sure you read up on this and other security features and keep them in mind. Later on it might be pretty difficult and laborious to redesign everything if it's built as a ring-0-only, everyone-is-superuser creative mode playground.
Question: I am trying to change the background color of st.text_input() box but unable to do so. I am not from web/app development background or with any html css skills so please excuse my naive or poor understanding in this field. So far I have tried: using this link test_color = st.write('test color') def text_input_color(url): st.markdown( f'<p style="background-color:#0066cc;color:#33ff33;">{url}</p>', unsafe_allow_html=True ) text_input_color("test_color")  Above code works on st.write() but not on st.text_input() I have also come across this link so using this approach and I have modified css for Textinput instead of stForm but this also didn't work and I am not sure what id to use for text input css=""" <style> [data-testid="stTextinput"] { background: LightBlue; } </style> """  Below is the inspect element screenshot of the webapp: 

Answer: I wouldn't rely on classes like .st-bd, .st-bb, or .st-b7 they are dynamically generated by Streamlit and can change (versions or runs). I would rather use aria-label.  import streamlit as st st.markdown(""" <style> .stTextInput input[aria-label="test color"] { background-color: #0066cc; color: #33ff33; } .stTextInput input[aria-label="test color2"] { background-color: #cc0066; color: #ffff33; } </style> """, unsafe_allow_html=True) st.text_input("test color") st.text_input("test color2")
Question: Is there any way optimizing the performance speed of this function? def func(X): n, p = X.shape R = np.eye(p) delta = 0.0 for i in range(100): delta_old = delta Y = X @ R alpha = 1. / n Y2 = Y**2 Y3 = Y2 * Y W = np.sum(Y2, axis=0) transformed = X.T @ (Y3 - (alpha * Y * W)) U, svals, VT = np.linalg.svd(transformed, full_matrices=False) R = U @ VT # is used as a stopping criterion delta = np.sum(svals) return R  Naively, I thought using numba would help because of the loop (the actual number of loops is higher), from numba import jit @jit(nopython=True, parallel=True) def func_numba(X): n, p = X.shape R = np.eye(p) delta = 0.0 for i in range(100): delta_old = delta Y = X @ R alpha = 1. / n Y2 = Y**2 Y3 = Y2 * Y W = np.sum(Y2, axis=0) transformed = X.T @ (Y3 - (alpha * Y * W)) U, svals, VT = np.linalg.svd(transformed, full_matrices=False) R = U @ VT delta = np.sum(svals) # is used as a stopping criterion return R  but to my surprise the numbaized function is actually slower. Why is numba not more effective in this case? Is there another option for me (preferably using numpy)? Note: You can assume X to be a "tall-and-skinny" matrix. MWE import numpy as np size = (10_000, 15) X = np.random.normal(size=size) %timeit func(X) # 1.28 s %timeit func_numba(X) # 2.05 s 

Answer: I tried to rewrite some stuff to speed things up. The only things I changed (apart from some formatting maybe) were removing the computation of the unused delta, pulling the transposition of X out of the loop as well as factorizing out the multiplication with Y for the computation of transformed which results in one fewer multiplications. def func2(X): n, p = X.shape R = np.eye(p) alpha = 1. / n XT = X.T for i in range(100): Y = X @ R Y2 = Y**2 W = np.sum(Y2, axis=0) transformed = XT @ (Y * (Y2 - (alpha * W))) U, svals, VT = np.linalg.svd(transformed, full_matrices=False) R = U @ VT return R  When I compare this func2 it to the function func as follows X = np.random.normal(size=(10_000, 15)) assert np.allclose(func(X), func2(X)) %timeit func(X) %timeit func2(X)  I get a speedup of more than 1.5x (it's not always as nice as that, however) 197 ms ± 44.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 127 ms ± 21.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Question: Suppose you have this dataframe d = {'date':['2019-08-25', '2019-09-01', '2019-09-08'], 'data':[31, 31, 31]} df_sample = pd.DataFrame(data=d) df_sample.head()  and you want to measure how much new data comes in on average each week. For example, we had 31 new rows on 8/25 and then on 9/1 we got an additional 31 rows so thats like a 100% increase. What I want to know is on average from one week to the next how much new data comes in? I know there is diff() and pct_change() but since this will just be 0 in these 3 samples I am wondering what would be the better approach here.

Answer: If your data is cumulative, you need a cumsum before pct_change: df_sample['change'] = df_sample['data'].cumsum().pct_change().mul(100)  Output:  date data change 0 2019-08-25 31 NaN 1 2019-09-01 31 100.0 2 2019-09-08 31 50.0  Intermediate:  date data cumsum change 0 2019-08-25 31 31 NaN 1 2019-09-01 31 62 100.0 2 2019-09-08 31 93 50.0  week to week If you want a week to week change, go with shift: df_sample['change'] = df_sample['data'].div(df_sample['data'].shift()).mul(100)  Output:  date data change 0 2019-08-25 31 NaN 1 2019-09-01 31 100.0 2 2019-09-08 31 100.0
Question: I am trying to count the number of separate events in a Dataframe (not number of occurences) Let's say i have this dataframe : df = pd.DataFrame([1, 2, 2, 2, 1, 1, 1, 1, 2, 1], columns=["events"])  And i want to count the number of separate events. If i use df.value_counts()  It' going to give me the number of occurences of each event : events 1 6 2 4 Name: count, dtype: int64.  But i want to count the number of separate events regardless of their lenght. The result should be something like this :  events 1 3 2 2  I wanted to know if there is a built in function for this. Any help or advice would be greatly appreciated. Thank you

Answer: Craft a mask (with shift and ne) to remove the successive duplicates and value_counts: df.loc[df['events'].ne(df['events'].shift()), 'events'].value_counts()  Output: events 1 3 2 2 Name: count, dtype: int64
Question: I wrote the following basic code sample that creates a main window with a button. When the button is pressed a second window with a progress bar should appear and stay open until the progress is complete. This works fine on Linux but on Windows the second window appears blank with no progress bar. Given that this will be part of an application that runs on Windows. What should I do? Is this a bug or what's going on? I'm using python 3.12.5 on both Linux and Windows 11 (build 22631). The code import tkinter.ttk as ttk import tkinter as tk import time def cpw(): tasks = 5 increment = 100 / tasks pw = tk.Toplevel() pw.title("Progress Window") pw.geometry("300x135") pb = ttk.Progressbar( pw, name="pb", orient=tk.HORIZONTAL, length=200, mode="determinate", ) pb.pack() i = 0 while i < tasks: i += 1 pb["value"] += increment pb.update_idletasks() time.sleep(1) pw.destroy() root = tk.Tk() root.title("Example") root.geometry("150x100") tk.Button(root, text="Process", command=cpw).pack() root.mainloop() 

Answer: Thanks to @acw1668 who pointed out in the comments that .update_idletasks() may not handle pending creation of widgets. Adding pb.update() or also pb.wait_visibility(pw) before pb.pack() will ensure the creation and visibility of the progress window and bar. Documentation here
Question: How do I manage that an input field value is empty, if it is not shown. In the example the text caption field is empty and not shown. If I show it by ticking "Show text caption field" and enter any text, the text appears in the output field. If I then untick "Show text caption field" the output field should also be empty again without having to manually. Not in general of course, but for some use cases this is quite important. from shiny import App, Inputs, Outputs, Session, render, ui app_ui = ui.page_fluid( ui.input_checkbox("show", "Show text caption field", False), ui.panel_conditional( "input.show", ui.input_text("caption", "Caption:"), ), ui.output_text_verbatim("value"), ) def server(input: Inputs, output: Outputs, session: Session): @render.text def value(): return input.caption() app = App(app_ui, server) 

Answer: Here you can extend the render.text such that the displayed value of the ui.output_text_verbatim is input.caption() if input.show() else "": from shiny import App, Inputs, Outputs, Session, render, ui app_ui = ui.page_fluid( ui.input_checkbox("show", "Show text caption field", False), ui.panel_conditional( "input.show", ui.input_text("caption", "Caption:"), ), ui.output_text_verbatim("value"), ) def server(input: Inputs, output: Outputs, session: Session): @render.text def value(): return input.caption() if input.show() else "" app = App(app_ui, server)