eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When using the jax_array.at[idx] function, I wish to be able to set values at both a set of specified rows and columns within the jax_array to another jax_array containing values in the same shape. For example, given a 5x5 jax array, I might want to set the values, jax_array.at[[0,3],:][:,[1,2]] to some 2x2 array of values. However, I am coming across an issue where the _IndexUpdateRef' object is not subscriptable. I understand the idea of the error (and I get a similar one when using 2 chained .at[]s), but I want to know if there is anyway to achieve the desired functionality within 1 line.

Answer: JAX follows the indexing semantics of NumPy, and NumPy's indexing semantics allow you to do this via broadcasted arrays of indices (this is discussed in Integer array indexing in the NumPy docs). So for example, you could do something like this: import jax.numpy as jnp x = jnp.zeros((4, 6), dtype=int) y = jnp.array([[1, 2], [3, 4]]) i = jnp.array([0, 3]) j = jnp.array([1, 2]) # reshape indices so they broadcast i = i[:, jnp.newaxis] j = j[jnp.newaxis, :] x = x.at[i, j].set(y) print(x)  [[0 1 2 0 0 0] [0 0 0 0 0 0] [0 0 0 0 0 0] [0 3 4 0 0 0]]  Here the i index has shape (2, 1), and the j index has shape (1, 2), and via broadcasting rules they index a 2x2 noncontiguous subgrid of the array x, which you can then set to the contents of y in a single statement.
Question: I'm trying to implement discriminated unions in Pydantic to select the correct class based on user input using the discriminator parameter. While the documentation suggests creating a nested model class to handle this easily, I'd like to use this functionality without introducing an additional nested model and have a similar behaviour as a normal Pydantic BaseModel class. I've tried to use RootModel as a workaround, but the resulting object is encapsulated within the .root property, which isn't ideal for my use case. I am able to do .model_dump() but unable to access the attributes on it directly. Is there a better way to implement this without creating a nested model or using RootModel? from typing import Literal, Union, Annotated from pydantic import BaseModel, Field, RootModel class Cat(BaseModel): pet_type: Literal["cat"] meows: int class Dog(BaseModel): pet_type: Literal["dog"] barks: float class Lizard(BaseModel): pet_type: Literal["reptile", "lizard"] scales: bool Animal = Annotated[ Union[Cat, Dog, Lizard], Field(discriminator="pet_type"), ] AnimalModel = RootModel[Animal] animal = AnimalModel.model_validate({"pet_type": "cat", "meows": 3}) try: # want to access the attributes directly print(animal.pet_type) except AttributeError as e: print(e) #> "RootModel[Annotated[Union[Cat, Dog, Lizard], FieldInfo(annotation=NoneType, required=True, discriminator='pet_type')]]" object has no attribute 'pet_type' # have to access the attributes by first accessing the .root attribute print(animal.root.pet_type)  (am using Pydantic v2.7)

Answer: You can use TypeAdapter instead of RootModel: from typing import Literal, Union, Annotated from pydantic import BaseModel, Field, TypeAdapter class Cat(BaseModel): pet_type: Literal["cat"] meows: int class Dog(BaseModel): pet_type: Literal["dog"] barks: float class Lizard(BaseModel): pet_type: Literal["reptile", "lizard"] scales: bool Animal = Union[Cat, Dog, Lizard] AnimalAdapter: TypeAdapter[Animal] = TypeAdapter( Annotated[Animal, Field(discriminator="pet_type")] ) animal_1 = AnimalAdapter.validate_python({"pet_type": "cat", "meows": 3}) animal_2 = AnimalAdapter.validate_python({"pet_type": "dog", "barks": 2}) try: print(animal_1.pet_type) # ok print(animal_2.pet_type) # ok except AttributeError as e: print(e)
Question: I am trying to do basically the opposite of drop_nulls(). I want to keep all rows that have at least one null. I want to do something like (but I don't want to list all other columns): for (name,) in ( df.filter( pl.col("a").is_null() | pl.col("b").is_null() | pl.col("c").is_null() ) .select("name") .unique() .rows() ): print( f"Ignoring `{name}` because it has at least one null", file=sys.stderr, ) df = df.drop_nulls() 

Answer: It sounds like you are looking for pl.Expr.any_horizontal. The following will keep all rows containing at least one null value (in any of the columns). df.filter(pl.any_horizontal(pl.all().is_null()))
Question: What is the most performant, idiomatic way of mapping over arrays of functions in JAX? Context: This GitHub issue shows a way to apply vmap to several functions using lax.switch. The example is reproduced below: from jax import lax, vmap import jax.numpy as jnp def func1(x): return 2 * x def func2(x): return -2 * x def func3(x): return 0 * x functions = [func1, func2, func3] index = jnp.arange(len(functions)) x = jnp.ones((3, 5)) vmap_functions = vmap(lambda i, x: lax.switch(i, functions, x)) vmap_functions(index, x) # DeviceArray([[ 2., 2., 2., 2., 2.], # [-2., -2., -2., -2., -2.], # [ 0., 0., 0., 0., 0.]], dtype=float32)  My specific questions are:  Is this (currently) the most idiomatic way of mapping over arrays of functions in JAX? What performance penalties, if any, does this method incur? (This refers to both runtime and/or compile-time performance.) 

Answer: For the kind of operation you're doing, where the functions are applied over full axes of an array in a way that's known statically, you'll probably get the best performance via a simple Python loop: def map_functions(functions: list[Callable[[Array], Array], x: Array) -> Array: assert len(functions) == x.shape[0] return jnp.array([f(row) for f, row in zip(functions, x)])  The method based on switch is designed for the more general case where the structure of the indices is not known statically.  What performance penalties, if any, does this method incur? (This refers to both runtime and/or compile-time performance.)  vmap of switch is implemented via select, which will compute the output of each function for the full input array before selecting just the pieces needed to construct the output, so if the functions are expensive to compute, it may lead to longer runtimes.
Question: I have 2 data sets: The first one describes what I expect: expected = { "name": ["start", "stop", "start", "stop", "start", "stop", "start", "stop"], "description": ["a", "b", "c", "d", "e", "f", "g", "h"], }  and the second one describes what I observe: observed = { "name": ["start", "stop", "start", "stop", "stop", "stop", "start"], "time": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7], }  I want to match all my observations to descriptions based on the order I expect. But once I see an inconsistency, nothing should match anymore. I managed to find the first inconsistency like: observed_df = pl.DataFrame(observed).with_row_index() expected_df = pl.DataFrame(expected).with_row_index() result = observed_df.join(expected_df, on=["index", "name"], how="left").select( "description", "time" ) """ ┌─────────────┬──────┐ │ description ┆ time │ │ --- ┆ --- │ │ str ┆ f64 │ ╞═════════════╪══════╡ │ a ┆ 0.1 │ │ b ┆ 0.2 │ │ c ┆ 0.3 │ │ d ┆ 0.4 │ │ null ┆ 0.5 │ -> First inconsistency gets a "null" description │ f ┆ 0.6 │ │ g ┆ 0.7 │ └─────────────┴──────┘ """  How can I propagate this null passed the first inconsistency? Also, my real data has an additional id column, where each id is a case like described above, and independent from other ids. Is it possible to somehow "group by id" and apply this logic all at once instead of working with each id separately: observed = { "id": [1, 2, 1, 2, 2], "name": ["start", "start", "stop", "stop", "stop"], "time": [0.1, 0.2, 0.3, 0.4, 0.5], } expected = { "id": [1, 1, 2, 2], "name": ["start", "stop", "start", "stop"], "description": ["a", "b", "c", "d"], } result = { "id": [1, 2, 1, 2, 2], "description": ["a", "c", "b", "d", None], "time": [0.1, 0.2, 0.3, 0.4, 0.5], } 

Answer: The check whether any null value appeared in an increasing window can be done using a cumulative evaluation, such as pl.Expr.cum_sum. A when-then-otherwise construct can be used to propagate null values accordingly. In your example, this might look as follows. ( observed_df .join( expected_df, on=["index", "name"], how="left", ) .select("description", "time") .with_columns( pl.when( pl.col("description").is_null().cum_sum() == 0 ).then( "description" ) ) )  shape: (7, 2) ┌─────────────┬──────┐ │ description ┆ time │ │ --- ┆ --- │ │ str ┆ f64 │ ╞═════════════╪══════╡ │ a ┆ 0.1 │ │ b ┆ 0.2 │ │ c ┆ 0.3 │ │ d ┆ 0.4 │ │ null ┆ 0.5 │ │ null ┆ 0.6 │ │ null ┆ 0.7 │ └─────────────┴──────┘  If you'd like to evaluate this expression separately for each group defined by id, a window function, such as pl.Expr.over, might be used. ... pl.when( pl.col("description").is_null().cum_sum() == 0 ).then( "description" ).over("id") # <-- ...
Question: Have: import polars as pl df = pl.DataFrame({'col1': [1,2,3], 'col2': [1, None, None]})  in polars dataframes, those Nones become nulls: > df ┌──────┬──────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞══════╪══════╡ │ 1 ┆ 1 │ │ 2 ┆ null │ │ 3 ┆ null │ └──────┴──────┘  Want: some command that returns the last two rows of df, since 2 & 3 are not null Tried: ..., but everything I've thought to try seems to drop/ignore records where one column is null:  df.filter(pl.col('col1')!=pl.col('col2')) # returns no rows df.filter(~pl.col('col1')==pl.col('col2')) # returns no rows df.filter(~pl.col('col1').eq(pl.col('col2'))) # returns no rows ... 

Answer: It is mentioned somewhat at the end of the .filter() docs. There are "missing" functions:  .eq_missing() .ne_missing()  df.filter(pl.col.col1.ne_missing(pl.col.col2))  shape: (2, 2) ┌──────┬──────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞══════╪══════╡ │ 2 ┆ null │ │ 3 ┆ null │ └──────┴──────┘
Question: #!/usr/bin/env python3 import unittest from unittest.mock import patch class User(object): def __init__(self): self.__name = None self.__authorised_users = ["me", "you"] local_input = input("please provide your windows 8 character lower case login: ") if local_input not in self.__authorised_users: raise ValueError("you have no permission to run this app") else: self.__name = local_input class TestUser(unittest.TestCase): def testUserClassFound(self): self.assertNotIsInstance(ModuleNotFoundError, User) @patch('builtins.input', lambda *args:"y") def testUserClassInit(self): # just check if user class has __name set to none local_object = User() self.assertEqual(local_object._User__name, None) if __name__ == "__main__": unittest.main()  I would like to, in the second test, just assure when the class object is created, the tester checks the class has the attribute __name and set to None. I need to patch the raise ValueError from the class init , but I can't find the correct patch.

Answer: You can patch User.__init__ with a wrapper that suppresses ValueError: def ignore(func, exception): def wrapper(*args, **kwargs): try: return func(*args, **kwargs) except exception: pass return wrapper class TestUser(unittest.TestCase): @patch('builtins.input', lambda *args:"y") @patch.object(User, '__init__', ignore(User.__init__, ValueError)) def testUserClassInit(self): local_object = User() self.assertEqual(local_object._User__name, None)  Demo: https://ideone.com/TEpHsF
Question: I have two files t.py: import functools import traceback def wrapper(func): @functools.wraps(func) def wrapped(*args, **kwargs): try: return func(*args, **kwargs) except Exception as e: traceback.print_exception(e) return wrapped @wrapper def problematic_function(): raise ValueError("Something went wrong")  and t2.py: from t import problematic_function problematic_function()  when I call from command line python t2.py, the stack trace information that has info from t2.py is lost when printing traceback.print_exception(e) in the t.py. What I get is Traceback (most recent call last): File "/home/c/t.py", line 9, in wrapped return func(*args, **kwargs) File "/home/c/t.py", line 18, in problematic_function raise ValueError("Something went wrong") ValueError: Something went wrong  where as if I remove the decorator I get: Traceback (most recent call last): File "/home/c/t2.py", line 3, in <module> problematic_function() File "/home/cu/t.py", line 17, in problematic_function raise ValueError("Something went wrong") ValueError: Something went wrong  How do I get the full stack trace in the wrapped function as without the wrapper? Thanks!

Answer: To print the full stack trace you can unpack traceback.walk_tb to obtain the last frame of the stack from the traceback, and pass it to traceback.print_stack as a starting frame to output the entire stack: def wrapper(func): @functools.wraps(func) def wrapped(*args, **kwargs): try: return func(*args, **kwargs) except Exception as e: *_, (last_frame, _) = traceback.walk_tb(e.__traceback__) traceback.print_stack(last_frame) return wrapped  Demo here
Question: I have a DataFrame that I am rounding. After the round, I subtract the original from the resultant. This gives me a data frame with a shape identical to the original, but which contains the amount of change the rounding operation caused. I need to transform this into a Boolean where there is a true flag for the max of the row, and everything else in the row is false. All steps but the final one are handled with a vectorized function. But I can't seem to figure out how to vectorize the last step. This is what I am currently doing: a = pd.DataFrame([[2.290119, 5.300725, 17.266693, 75.134857, 0.000000, 0.000000, 0.007606], [0.000000, 7.560276, 55.579175, 36.858266, 0.000000, 0.000000, 0.002284], [0.001574, 15.225538, 39.309742, 45.373800, 0.000951, 0.001198, 0.087197], [0.000000, 55.085390, 15.547927, 29.327661, 0.000000, 0.017691, 0.021331], [0.000000, 66.283488, 15.636673, 17.912315, 0.000000, 0.003185, 0.164339]]) b = a.round(-1) # round to 10's place (not 10ths) c = b-a round_modifier = c.apply(lambda x: x.eq(x.max()), axis="columns") print(round_modifier)   0 1 2 3 4 5 6 0 False False False True False False False 1 False False True False False False False 2 False True False False False False False 3 False True False False False False False 4 False False True False False False False  I am aware of DataFrame.idxmax(axis="columns"), which gives me the column name (of each row) where the max is found, but I can't seem to find a (pythonic) way to take that and populate the corresponding flag with a True. The lambda expression I'm using gives the correct result, but I'm hoping for a faster method. For anyone wondering, the use case is that I want to round the values in the original data frame to the tens place, such that they sum to 100. I have pre-scaled this data so it should be close, but the rounding can cause the sum to come to 90 or 110. I intend to use this T/F matrix to decide which rounded value caused the most delta, then round it in the opposite direction since this is the minimum impact method with which to coerce the series to properly sum to 100 in chunks of 10.

Answer: You can use idxmax to get the position of column with the max value, and use numpy broadcasting to match the position with the column. m = c.columns.to_numpy() == c.idxmax(axis=1).to_numpy()[:, None] new_df = pd.DataFrame(np.where(m, True, False), columns=c.columns)  End result:  0 1 2 3 4 5 6 False False False True False False False False False True False False False False False True False False False False False False True False False False False False False False True False False False False
Question: Suppose I am given a few numpy arrays, say a, b and c, which are assumed to be broadcastable. Is there a standard or othwerwise an elegant way to find the array shape after broadcasting? Of course, something like (a+b+c).shape would work, but is very inefficient if I am only interested in the shape of the result.

Answer: You can use broadcast_shapes. Like this result_shape = np.broadcast_shapes(a.shape, b.shape, c.shape)  Here is the documentation page: https://numpy.org/doc/stable/reference/generated/numpy.broadcast_shapes.html Good luck!
Question: I have a Python web application built with Plotly Dash and deployed on Azure App Service using Python 3.12. As Azure App Service has not yet enabled Python 3.12 version to have application insight enabled, I have utilized the package: azure-monitor-opentelemetry==1.6.2  within my application to log exceptions into my application insight resource. However, as I deploy my web application on Azure App Service, and when someone is accessing the web app, my application logs the following exception onto the application insight: Failed to derive Resource from Tracer Provider: 'ProxyTracerProvider' object has no attribute 'resource' Traceback (most recent call last): File "/tmp/8dcd22e385c2da5/antenv/lib/python3.12/site-packages/azure/monitor/opentelemetry/exporter/export/trace/_exporter.py", line 91, in export resource = tracer_provider.resource # type: ignore  AttributeError: 'ProxyTracerProvider' object has no attribute 'resource' I have implemented the azure-monitor-opentelemetry in my application simply as following: appopentelemetry.py: from azure.monitor.opentelemetry import configure_azure_monitor from models.environmentmanager.environmentmanager import EnvironmentManager def azure_monitoring_open_telemetry(): environment_manager = EnvironmentManager() connection_string = environment_manager.get_connection_string() # The app insight connection string return configure_azure_monitor( connection_string=connection_string, enable_live_metrics=True, )  and in my app.py: ...imports... environment_manager = EnvironmentManager() # This should be executed if the environment is not local, this is set in .env if ran locally # and set in environment variables of the app service on Azure when deployed if not environment_manager.get_is_local(): from functions.app.appopentelemetry import azure_monitoring_open_telemetry azure_monitoring_open_telemetry() ... app = Dash(__name__, use_pages=True, external_stylesheets=stylesheets) ... app.layout = dmc.MantineProvider(...) if __name__ == '__main__': app.run(debug=True, port=8000)  Can I know what I am doing wrong?

Answer: Version 1.6.2 also upgrades azure-monitor-opentelemetry-exporter to "1.0.0b29" and it has the braking changes. Just downgrade to azure-monitor-opentelemetry-exporter = "1.0.0b28" and maybe to azure-monitor-opentelemetry = "1.6.1". It will solve the issue for now. In general, I think that config should be done differntly rather then using configure_azure_monitor but it is not that clear from the documentation. Try to read here https://opentelemetry.io/docs/languages/python/getting-started/
Question: In the following code snippet the dataclass Derived is derived from dataclass Base. The Derived dataclass is setting new default values for field1 and field2. from dataclasses import dataclass @dataclass class Base: field1: str field2: str = "default_base_string" @dataclass class Derived(Base): field1 = "default_string1" field2 = "default_string2" print(Derived())  Because Derived is a dataclass and has set a default value for all member variables field1 and field2, I expected that I could simply initialize the class calling Derived(). However I get the following error: TypeError: Derived.__init__() missing 1 required positional argument: 'field1'  Is it not possible to set the default value for a member variable in a derived dataclass, so that this member variable becomes optional in the autogenerated __init__() method? Must I always define my own __init__() method in the Derived class? Or am I doing something wrong here?

Answer: To set default values in Derived you need to add the type annotation @dataclass class Derived(Base): field1: str = "default_string1" field2: str = "default_string2" print(Derived()) # Derived(field1='default_string1', field2='default_string2')
Question: I can ask my question best by just giving an example. Let's say I want to use a list comprehension to generate a set of 3-element tuples from two loops, something like this: [ (y+z,y,z) for y in range(10) if y%2==0 for z in range(20) if z%3==0 ]  This works, giving me [(0, 0, 0), (3, 0, 3), (6, 0, 6), (9, 0, 9), (12, 0, 12), (15, 0, 15), ... ]  I am wondering, though, if there is a way to do it more cleanly, something to the effect of [ (x,y,z) for y in range(10) if y%2==0 for z in range(20) if z%3==0 ... somehow defining x(y,z) ... ]  I would consider something like this to be more clean, especially since what I really need to do is much more complicated than the example I give here. Everything I have tried has given me a syntax error.

Answer: You can do: out = [ (x, y, z) for y in range(10) if y % 2 == 0 for z in range(20) if z % 3 == 0 for x in [y + z] # <-- initialize `x` in list-comprehension ]  This is optimized since Python 3.9: https://docs.python.org/3/whatsnew/3.9.html#optimizations
Question: At my workplace pip is not able to access the outside world to download packages. I'm not sure what system exactly is preventing this, but ideally I shouldn't be installing any old packages from the internet anyway. The only way I can install packages from online is to download a source distribution or wheel from the PyPI site or from the package's github and run pip install --no-deps <package path> so that pip doesn't attempt to go online for dependencies which would just hang for a long time and then fail. I am developing some tools for my own use, and would like to package and distribute them to other members of my team so that they can be installed from a tarball or wheel, and also not require them to go online and pull more dependencies manually. The tools I'm writing are using the standard library so they don't require any online dependencies. I'm leaning towards disregarding deprecation and using the setuptools versions of the now-removed distutils module, and writing a setup.py file for my package. However, this is deprecated or at least seems heavily discouraged. The setuptools user guide gives instructions for using the build package, but this package has a set of dependencies, and some of those have their own dependencies, and manually installing all of those packages by hand is a serious headache. At the top-level build requires:  colorama (this feels like it should be optional) importlib-metadata packaging pyproject_hooks tomli  and some of these have their own list of dependencies. Does anyone in a similarly restricted environment have a preferred method for building and installing internal packages? If so, what is your process? In an ideal world I could just install all of the dependencies I need with no problem, but I understand that has security implications and I don't think I'll be able to make any sort of exceptions for pip's access being blocked. What I've Tried I have read documentation for various Python packaging methods, and the simplest I could find that isn't "deprecated" still requires the build package, which as I've said requires a large-enough tree of dependencies that installing them all manually is out of the question and still is in a gray area of acceptability for security reasons. The setuptools documentation recommends against the use of setup.py: setuptools quickstart The latest version of Python officially removes distutils from the standard library and continuing to use setup.py builds with modules from setuptools seems to be discouraged for the most part PEP 0632

Answer: In my situation it seems that the most straightforward course of action is to ignore the fact that directly using setup.py is discouraged and do it anyway. With setup.py importing setup from setuptools I'm able to build a package that can be installed with pip by running python setup.py sdist Of the suggestions in the comments this seems to be only way to build a package that only requires installation of setuptools if it isn't already installed. I would still like a more modern and elegant solution, but that may not be possible due to whatever is restricting pip's access to the internet.