eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: This code works: cohort = r'priority' result2025 = df.groupby([cohort],as_index=False).agg({'resolvetime': ['count','mean']})  and this code works cohort = r'impactedservice' result2025 = df.groupby([cohort],as_index=False).agg({'resolvetime': ['count','mean']})  and this code works result2025 = df.groupby(['impactedservice','priority'],as_index=False).agg({'resolvetime': ['count','mean']})  but what is not working for me is defining the cohort variable to be cohort = r'impactedservice,priority' # a double-cohort result2025 = df.groupby([cohort],as_index=False).agg({'resolvetime': ['count','mean']})  That gives error: KeyError: 'impactedservice,priority'  How to properly define the cohort variable in this case?

Answer: The issue is that when you do: cohort = r'impactedservice,priority'  You're creating a single string, not a list of column names. Pandas treats that as a single column name (which doesn’t exist), hence the KeyError. Correct way: Define cohort as a list of column names: cohort = ['impactedservice', 'priority'] result2025 = df.groupby(cohort, as_index=False).agg({'resolvetime': ['count', 'mean']})  Now groupby knows you're grouping by multiple columns. You can build cohort dynamically as a list too if needed: cohort = ['impactedservice'] if use_priority: cohort.append('priority')
Question: Im having trouble setting the y axis to start at zero for the following shiny express python script. Instead it starts at 4.1 . the set_ylim is having no affect. from shiny.express import input, render, ui import matplotlib.pyplot as plt import numpy as np data = { "Maturity": ['1Y', '2Y', '3Y', '4Y', '5Y', '6Y', '7Y', '8Y'], "Yield": [4.1, 4.3, 4.5, 4.7, 4.8, 4.9, 5.0, 5.1] } data=np.array([data["Maturity"], data["Yield"]]) #df = pd.DataFrame(data) print(data[1]) def create_line_plot(): x = data[0] y = data[1] fig, ax = plt.subplots() #fig.title("Yield Curve") ax.bar(x, y) ax.set_xlabel("Maturity") ax.yaxis.set_label_text("Yield (%)") ax.set_ylim(bottom=0) # Ensures the y-axis starts at 0 # Ensures the y-axis starts at 0 return fig @render.plot def my_plot(): return create_line_plot() 

Answer: This is an issue which results from how you process your data. Avoid the array conversion: from shiny.express import render import matplotlib.pyplot as plt data = { '1Y': 4.1, '2Y': 4.3, '3Y': 4.5, '4Y': 4.7, '5Y': 4.8, '6Y': 4.9, '7Y': 5.0, '8Y': 5.1 } @render.plot def create_line_plot(): x = list(data.keys()) y = list(data.values()) fig, ax = plt.subplots() ax.bar(x, y) ax.set_xlabel("Maturity") ax.yaxis.set_label_text("Yield (%)") ax.set_ylim(bottom=0) return fig
Question: How can I call task2 from func without declaring func async and awaiting it? My first thought was to create a thread and use run_coroutine_threadsafe but it deadlocks. Same as not using a thread. Do I have to start a new loop? import asyncio from threading import Thread async def task2(): print("starting task2...") await asyncio.sleep(1) print("finished task2.") return "done" def func(loop=None): print("running func...") if not loop: loop = asyncio.get_running_loop() assert loop future = asyncio.run_coroutine_threadsafe( task2(), loop) result = future.result() print(f"{result=}") print("done func...") async def task1(): print("starting task1...") await asyncio.sleep(1) # func() loop = asyncio.get_running_loop() t = Thread(target=func, args=(loop,)) t.start() t.join() print("finished task1.") if __name__ == '__main__': asyncio.run(task1()) 

Answer: python threading synchronization primitives such as Thread.join don't work well with asyncio because they suspend the thread and therefore block the running eventloop, so run_coroutine_threadsafe cannot use the blocked eventloop. Instead you have loop.run_in_executor for creating threaded tasks that don't block the eventloop. import asyncio async def task2(): print("starting task2...") await asyncio.sleep(1) print("finished task2.") return "done" def func(loop: asyncio.AbstractEventLoop): print("running func...") future = asyncio.run_coroutine_threadsafe( task2(), loop) result = future.result() print(f"{result=}") print("done func...") async def task1(): print("starting task1...") await asyncio.sleep(1) loop = asyncio.get_running_loop() task = loop.run_in_executor(None, func, loop) await task # doesn't block the eventloop print("finished task1.") if __name__ == '__main__': asyncio.run(task1())  starting task1... running func... starting task2... finished task2. result='done' done func... finished task1.  Each evenloop has a default ThreadPoolExecutor that it uses if you pass None as the executor, but it has a limited number of workers and is intended for computational and non-blocking work, while you are using it for blocking work, so it may be beneficial to override it with a larger number of workers using loop.set_default_executor, or you can have each part of your codebase use a different ThreadPoolExecutor with a different size to avoid one part hogging all the workers with blocking tasks. threads are created lazily anyway.  Another solution is to use asyncio.run which creates a new eventloop on the current thread to run this coroutine. just make sure you only call it on a thread that doesn't already have an eventloop running or it will throw an exception. I think sending back the coroutine to the original loop as you are doing is a good solution, and is actually the most performant, otherwise you can create a shared daemon thread that has an eventloop just for the sake of sending coroutines to it.
Question: As far as I can tell, the following MRE conforms to the relevant documentation: import polars df = polars.read_excel( "/Volumes/Spare/foo.xlsx", engine="calamine", sheet_name="natsav", read_options={"header_row": 2}, columns=(1,2,4,5,6,7), # columns 0 and 3 are not needed ) print(df.head())  The issue here is that the documentation states that for the columns parameter:  Columns to read from the sheet; if not specified, all columns are read. Can be given as a sequence of column names or indices.  Clearly, a tuple is a sequence. However, running this code results in an exception as follows: _fastexcel.InvalidParametersError: invalid parameters: `use_columns` callable could not be called (TypeError: 'tuple' object is not callable)  Further research reveals that the required callable should return bool. So: def colspec(c): print(type(c)) return True  I then change the read_excel call to include columns=colspec. The program now runs without exception and reveals that the parameter passed is a class of type builtins.ColumnInfoNoDtype. Unfortunately, I can't find any documentation for that type. Is the documentation wrong? How is one supposed to used polars.read_excel to load only certain specific columns?

Answer: When you use Calamine, which is the default engine and which you've specified explicitly, the docs say (emphasis mine):  this engine can be used for reading all major types of Excel Workbook (.xlsx, .xlsb, .xls) and is dramatically faster than the other options, using the fastexcel module to bind the Rust-based Calamine parser.  This corresponds with what the error message tells you: _fastexcel.InvalidParametersError: invalid parameters: `use_columns` callable could not be called (TypeError: 'tuple' object is not callable) ^^^^^^^^^^ ^^^^^^^^^^^  This isn't an error from read_excel itself; the columns argument is being passed through to fastexcel. Looking at fastexcel's docs, we can see the use_columns parameter defined as:  use_columns: Union[list[str], list[int], str, Callable[[ColumnInfoNoDtype], bool], NoneType] = None,  (This is also where the ColumnInfoNoDtype type is defined, it's again not part of polars itself.) and described as:  Specifies the columns to use. Can either be:  None to select all columns A list of strings and ints, the column names and/or indices (starting at 0) A string, a comma separated list of Excel column letters and column ranges (e.g. “A:E” or “A,C,E:F”, which would result in A,B,C,D,E and A,C,E,F) A callable, a function that takes a column and returns a boolean   Here we can see that, as well as the Callable[[ColumnInfoNoDtype], bool] type your input is apparently being interpreted as, there are list[int] and list[str] options - not Sequence, which would include tuple. Presumably the issue in the polars docs is that how exactly these things are going to be interpreted depends on which engine they're going to - they're trying to describe the general inputs, but in this case it seems that the engine is looking for a list specifically.
Question: I have written the following code that returns a solution. import sympy as sym import numpy as np z = sym.Symbol('z') e = sym.Symbol('e') f = sym.Function('f') edo = sym.diff(f(z), z , 2) + e * f(z) soln = sym.dsolve(edo, f(z)) print(soln.rhs)  the above code returns: C1*exp(-z*sqrt(-e)) + C2*exp(z*sqrt(-e))  I want to be able to access the elements of this 'soln.rhs' directly and plot them. I could copy and paste the results, but I want to do something like: plot(x, soln.rhs[0])  which would plot exp(-z*sqrt(-e)) The reason for this is that I am analysing many different types of ODE, some of which return solutions which are combinations of Airy functions like C1*airyai(-e + z) + C2*airybi(-e + z)  Does anyone know how to access the elements of the general solution? I have looked through the documentation and nothing really pops out.

Answer: What you are looking for is the args attribute of a Sympy's symbolic expression. For example: print(soln.rhs.args[0]) # C1*exp(-z*sqrt(-e)) print(soln.rhs.args[1]) # C2*exp(z*sqrt(-e))  You might also want to insert appropriate values for the integrations constants by using the subs method: C1, C2 = symbols("C1, C2") soln.rhs.subs({C1: 2, C2: 3}) # random numeric values to show how to do it.  Then, you can plot it: # plot the solution for z in [0, 10] plot(soln.rhs.subs({C1: 2, C2: 3}), (z, 0, 10))
Question: I am using matplotlib's Slider to do a simple dynamic plot of a sine curve. I want to change the frequency using a slider. The code looks like this: import numpy as np import matplotlib.pyplot as plt from matplotlib.widgets import Slider x = np.linspace(400, 800, 400) fig, ax = plt.subplots() fig.subplots_adjust(bottom=0.25) mod = 1. lambda_plot, = ax.plot(x, np.sin(mod*x*2*np.pi/500)) ax_mod_slider = fig.add_axes([0.3, 0.1, 0.5, 0.04]) mod_slider = Slider( ax = ax_mod_slider, label = "modulation", valmin = 0, valmax = 20, valinit = 1., orientation = "horizontal") def update_mod(val): mod = mod_slider.val redraw() fig.canvas.draw_idle() def redraw(): lambda_plot.set_ydata(np.sin(mod*x*2*np.pi/500)) mod_slider.on_changed(update_mod) plt.show()  This does only work if I put the code in redraw() directly in update_mod(), like this: def update_mod(val): mod = mod_slider.val lambda_plot.set_ydata(np.sin(mod*x*2*np.pi/500)) fig.canvas.draw_idle()  Why can I not call another function for changing the plot? In this example, I could put it all into update_mod(), but as I do more complicated calculations, I thought it might be good to be able to split update_mod() into separate functions.

Answer: In the redraw() function: def redraw(): lambda_plot.set_ydata(np.sin(mod * x * 2 * np.pi / 500))  You're using the variable mod, but this mod is not defined in the local scope of redraw(), nor is it properly declared as global or passed as an argument. So Python looks for it in the outer scope and finds the mod = 1. you defined at the beginning of the script which never changes. When update_mod() changes mod, it does so in its local scope, and that doesn't affect the mod that redraw() sees. Just pass the mod value and it should work properly: def redraw(mod): lambda_plot.set_ydata(np.sin(mod * x * 2 * np.pi / 500))
Question: I'm trying to understand how NumPy determines the dtype when creating an array with mixed types. I noticed that the inferred dtype for strings can vary significantly depending on the order and type of elements in the list. print(np.array([1.0, True, 'is'])) # Output: array(['1.0', 'True', 'is'], dtype='<U32') print(np.array(['1.0', True, 'is'])) # Output: array(['1.0', 'True', 'is'], dtype='<U5') print(np.array(['1.0', 'True', 'is'])) # Output: array(['1.0', 'True', 'is'], dtype='<U4')  I understand that NumPy upcasts everything to a common type, usually the most general one, and that strings tend to dominate. But why does the resulting dtype (<U32, <U5, <U4) differ so much when the content looks almost the same? Specifically:  Why does np.array([1.0, True, 'is']) result in <U32? What determines the exact length in the dtype (e.g., <U4 vs <U5)? Is there a consistent rule for how NumPy infers the dtype and string length in such cases? 

Answer: Looking at your array contents and the NumPy type promotion rules, I guess the following applies:  For some purposes NumPy will promote almost any other datatype to strings. This applies to array creation or concatenation.  This leaves us with the question about the string lengths. For the complete array, we need to choose a string length so that all values can be represented without loss of information. In your examples, the contents have the following data types:  Strings ('is', 'True', '1.0'): for these, NumPy just needs to reserve their actual length (thus, if there are multiple strings in the same array, the maximum length of all occurring strings). Booleans (True): for converting them to a string, NumPy reserves a string length of 5, since all possible converted values are 'True' (length 4) and 'False' (length 5). We can easily verify this: np.array(True).astype(str) # >>> array('True', dtype='<U5')   Floats (1.0): for converting them to a string, NumPy reserves a string length of 32. I assume this is for round-trip safety (i.e. to get the exact same value when converting the string representation back to float). I would have expected that a shorter length (somewhere between 20 and 30) should be enough, but maybe 32, a power of 2, has been chosen for better memory alignment properties. In any case, again, we can verify this: np.array(1.).astype(str) # >>> array('1.0', dtype='<U32')    Now to your examples:  np.array([1.0, True, 'is']): we have a float 1.0 (→ string length 32), a boolean True (→ string length 5), and a string 'is' of length 2: The maximum length to represent all values is 32. np.array(['1.0', True, 'is']): we have a string '1.0' of length 3, a boolean True (→ string length 5), and a string 'is' of length 2: The maximum length to represent all values is 5. np.array(['1.0', 'True', 'is']): we have a string '1.0' of length 3, a string 'True' of length 4, and a string 'is' of length 2: The maximum length to represent all values is 4.
Question: Is there a simple syntax for creating references to an arbitrary number of neighbouring array elements in numpy? The syntax is relatively straightforward when the number of neighbours is hard-coded. A stencil width of three for example is import numpy as np x = np.arange(8) # Hard-coded stencil width of 3 x_neighbours = ( x[ :-2] x[ 1:-1] x[ 2: ] )  However, my attempt at arbitrary width stencils is not particularly readable  nStencil = 3 x_neighbours = ( x[indexStart:indexStop] for indexStart, indexStop in zip( (None, *range(1,nStencil)), (*range(1-nStencil,0), None), ) )  Is there a better approach?

Answer: I'd recommend using sliding_window_view. Change: nStencil = 3 x_neighbours = ( x[indexStart:indexStop] for indexStart, indexStop in zip( (None, *range(1,nStencil)), (*range(1-nStencil,0), None), ) )  To: nStencil = 3 sliding_view = sliding_window_view(x, nStencil) x_neighbours = tuple(sliding_view[:, i] for i in range(nStencil))
Question: I want to explain to Pyright that my variables in class and instance have different types. I managed to overload __get__ method to achieve this, but now Pyright complains about initialization of instances (see last line): Literal[1] is not assignable to Field[int] My code: import typing as ty from dataclasses import dataclass class Field[InstanceT]: def __init__(self, default: InstanceT): self.default = default self.first = True def __get__(self, obj, owner): if self.first: self.first = False return self.default return self if ty.TYPE_CHECKING: @ty.overload def __get__(self, obj: None, owner: type) -> ty.Self: ... @ty.overload def __get__(self, obj: object, owner: type) -> InstanceT: ... @dataclass class Model: field: Field[int] = Field(0) if __name__ == "__main__": # It`s fine class_field: Field = Model.field instance_field: int = Model().field assert isinstance(class_field, Field) assert isinstance(instance_field, int) # Literal[1] is not assignable to field[int] obj = Model(field=1)  Asserts are true, but Pyright complains.

Answer: You want to have a data-descriptor, such it needs a __set__ method. You will get an error depending on the signature of __set__, you want it to accept your generic. A working example could look like this, the instance value will be stored on the objects _field attribute, see the __set_name__ magic, you could of course also store in the Field and not on the instance. I am not sure about your self.first logic - so you might want to change some parts. import typing as ty from dataclasses import dataclass class Field[InstanceT]: def __init__(self, default: InstanceT): self.default = default self.first = True @ty.overload def __get__(self, obj: None, owner: type) -> ty.Self: ... @ty.overload def __get__(self, obj: object, owner: type) -> InstanceT: ... def __get__(self, obj, owner): if self.first: self.first = False return self.default if obj is None: # <-- called on class return self return getattr(obj, self._name, self.default) # <-- called on instance def __set_name__(self, owner, name): self._name = "_" +name def __set__(self, obj, value: InstanceT): setattr(obj, self._name, value) @dataclass class Model: field: Field[int] = Field(0) if __name__ == "__main__": class_field = Model.field reveal_type(class_field) # Field[int] model = Model() instance_field: int = model.field reveal_type(instance_field) # int assert isinstance(class_field, Field) assert isinstance(instance_field, int) # Literal[1] is not assignable to field[int] obj = Model(field=1) # OK Model(field="1") # Error
Question: This Python 3.12.7 script with Numpy 2.2.4: import numpy as np, timeit as ti a = np.random.rand(1000).astype(np.float32) print(f'Minimum, median and maximum execution time in us:') for fun in ('np.fabs(a)', 'np.abs(a)'): t = 10**6 * np.array(ti.repeat(stmt=fun, setup=fun, globals=globals(), number=1, repeat=999)) print(f'{fun:20} {np.amin(t):8,.3f} {np.median(t):8,.3f} {np.amax(t):8,.3f}')  produces these results on AMD Ryzen 7 3800X: Minimum, median and maximum execution time in us: np.fabs(a) 1.813 1.843 4.929 np.abs(a) 0.781 0.811 1.463  indicating that np.fabs() is more than 2x slower than np.abs(), despite the latter having more functionality. What is the reason?

Answer: fabs always calls the C math library function of the same name (or in this case, the fabsf type variation). Therefore the operation cannot be inlined or vectorized. I have verified this by injecting a custom version using LD_PRELOAD. I've checked the source code of glibc (which just calls __builtin_fabsf(x)) and looked at the generated code with godbolt. I see no complexity (e.g. NaN handling or math exceptions) that differentiate fabs from the fastest, simple abs implementation. I assume numpy always call the library in the f… functions out of principle. Similar effects can be expected from fmin and fmax, for example (though here the implementation is actually more complex than plain min() and max(). From looking at old bug reports, it appears that the performance difference (and the signaling math behavior) are actually platform-dependent. On MIPS, abs() used to be (or still is?) slower as the compiler could not turn the generic C code into a simple bit mask due to the potential of floating point exceptions (which fabs should never do). This also highlights that implementing fabs without compiler support is not as simple as writing x < 0 ? -x : x because -0 has to be differentiated from +0, which the comparison does not. np.abs seems to do the right thing on modern numpy, even though a simple C version would not but I have not investigated where and how the behavior is implemented for floating point types.
Question: I have a rather large app.py file, so I'd like to take a nav panel out and store it in a separate file. I'm not sure how to access code from a different file and include it in the main app. The app.py file: import page from shiny import reactive, render, req, ui from shiny.express import input, ui with ui.navset_hidden(id="selected_navset_tab"): # Homepage with ui.nav_panel("Welcome", value="page_home"): with ui.card(): # Variable calculated ui.input_selectize( "filler", "Filler", ["list", "of", "items", "here"], multiple=False )  The other file, page.py: from shiny import reactive, render, req, ui from shiny.express import input, ui with ui.nav_panel("Page I want to put in the main app", value="other_page"): @render.express def filler_text(): "Filler text"  How can I import the other_page nav panel to show as part of the navset_tab in the main app.py file, without actually putting it in the code?

Answer: You can wrap the other_page nav panel into a function and use this function inside the main app. However, doing this results in an empty additional nav panel and you need to apply express.expressify. This is because otherwise only the return value of the nav function is shown (None), where we instead want to display the result of each line (see the Programming UI section within the docs for more details). page.py from shiny import render from shiny.express import ui, expressify @expressify def otherNav(): with ui.nav_panel("Page I want to put in the main app", value="other_page"): @render.express def filler_text(): "Filler text"  app.py import page from shiny import ui from shiny.express import input, ui with ui.navset_tab(id="selected_navset_tab"): # Homepage with ui.nav_panel("Welcome", value="page_home"): with ui.card(): # Variable calculated ui.input_selectize( "filler", "Filler", ["list", "of", "items", "here"], multiple=False ) # other Nav page.otherNav()
Question: I have a pandas dataframe that has information about total attendance for schools grouped by School, District, Program, Grade, and Month #. The data looks like the following (df): School District Program Grade Month Count 123 456 A 9-12 10 100 123 456 B 9-12 10 95 321 654 A 9-12 10 23 321 456 A 7-8 10 40  Some of the counts are inflated and need to be reduced based on the data from another dataframe (ToSubtract): School District Program Grade Month Count 123 456 A 9-12 10 10 321 654 A 9-12 10 8  Both dataframes are already grouped so there will be no duplicate grouping. Subtracting ToSubtract from df will result in: School District Program Grade Month Count X 123 456 A 9-12 10 90 * 123 456 B 9-12 10 95 321 654 A 9-12 10 15 * 321 456 A 7-8 10 40  (X column to be marked with * to indicate value was modified). df has a lot more entries for all of the other schools, districts, months, etc. I was looking into df.sub() but it looks like the elements have to be lined up. My other idea was to use df.iterrows() to go through each row of df and check if there is a corresponding row in ToSubtract but this seems very inefficient. What would be the best way to subtract one dataframe from another, matching several columns

Answer: A possible solution: cols = ['School', 'District', 'Program', 'Grade', 'Month'] df1.set_index(cols).sub(df2.set_index(cols), fill_value=0).reset_index()  First, it defines the list cols with the columns used to match records (School, District, Program, Grade, and Month). Then, it sets these columns as the index for both dataframes using set_index, allowing pandas to align rows based on these keys. It uses sub to subtract df2 from df1, with fill_value=0 ensuring that missing rows in either dataframe are treated as zeros (i.e., subtracting from zero or not subtracting at all). Finally, it resets the index back to columns with reset_index. Output:  School District Program Grade Month Count 0 123 456 A 9-12 10 90.0 1 123 456 B 9-12 10 95.0 2 321 456 A 7-8 10 40.0 3 321 654 A 9-12 10 15.0