eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Using the small example below, I'm seeking to aggregate (sum) the values in the 3D dat_arr array using two other arrays to guide the grouping. The first index of dat_arr is related to time. The second and third indices are related to spatial (X, Y) locations. How can I sum values in dat_arr such that the temporal binning is guided by the contents of tim_idx (same length as the first dimension of dat_arr) and the spatial binning uses zon_arr (has same dimensions as the last two indices of dat_arr)? import numpy as np import matplotlib.pyplot as plt zon_arr = np.zeros((3,5)) tim_idx = np.array([0,0,1,1,2,2,3,3]) # set up arbitrary zones zon_arr[1, :3] = 1 zon_arr[1, 3:] = 2 # plt.imshow(zon_arr) # plt.show() # generate arbitrary array with data # first index = time; last 2 indices represent X-Y pts in space # last two indices must have same dims as zon_arr np.random.seed(100) dat_arr = np.random.rand(8, 3, 5)  So the output I'm expecting would give me the sum of values contained in dat_arr for each unique value in tim_idx and zon_arr. In other words, I would expect output that has 3 value (corresponding to 3 zones) for each of the 4 unique time values in tim_idx?

Answer: First, let's compute this with a loop to get a sense of the potential output: sums = {} # for each combination of coordinates for i in range(len(tim_idx)): for j in range(zon_arr.shape[0]): for k in range(zon_arr.shape[1]): # add the value to the (time, zone) key combination key = (tim_idx[i], zon_arr[j, k]) sums[key] = sums.get(key, 0) + dat_arr[i, j, k]  which gives us: {(0, 0): 8.204124414317336, (0, 1): 3.8075543426771645, (0, 2): 1.2233223229754382, (1, 0): 7.920231812858928, (1, 1): 4.150642040307019, (1, 2): 2.4211020016615836, (2, 0): 10.363684964675313, (2, 1): 3.06163710842573, (2, 2): 1.9547272492467518, (3, 0): 10.841595367423158, (3, 1): 2.6617183569891893, (3, 2): 2.0222766813453674}  Now we can leverage numpy indexing to perform the same thing in a vectorial way. meshgrid to generate the indexer, unique to get the unique combinations, and bincount to compute the sum per group: # create the indexer from the combination of time/zone i, j = np.meshgrid(tim_idx, zon_arr, indexing='ij') coord = np.c_[i.ravel(), j.ravel()] # alternatively # coord = np.c_[np.repeat(tim_idx, zon_arr.size), # np.tile(zon_arr.flat, len(tim_idx))] # identify the unique combinations for later aggregation keys, idx = np.unique(coord, return_inverse=True, axis=0) # compute the counts per key sums = np.bincount(idx, dat_arr.ravel())  Output: # keys array([[0, 0], [0, 1], [0, 2], [1, 0], [1, 1], [1, 2], [2, 0], [2, 1], [2, 2], [3, 0], [3, 1], [3, 2]]) # sums array([ 8.20412441, 3.80755434, 1.22332232, 7.92023181, 4.15064204, 2.421102 , 10.36368496, 3.06163711, 1.95472725, 10.84159537, 2.66171836, 2.02227668])
Question: I am working with a legacy system that generates delimited files (e.g., CSV), but the number of columns in these files is inconsistent across rows (ragged rows). I am reading the file from ADLS with Polars, but I'm encountering an issue depending on the structure of the second row in the file. pl.read_csv('sample.csv', has_header=False, skip_rows=1, infer_schema=False, infer_schema_length=None, ignore_errors=True)   If the second row has more columns than subsequent rows, Polars reads the file successfully and fills the missing values in subsequent rows with null. However, if subsequent rows have more columns than the second row, I get the following exception  ComputeError: found more fields than defined in 'Schema' Consider setting 'truncate_ragged_lines=True'. Is there a way to handle such cases dynamically in Polars, or do I need to preprocess the file to fix these inconsistencies before reading? Any alternative approaches or solutions to this problem would be appreciated! Example Data - Failure ID,Name,Age 1,John,28 2,Jane,35,California,USA 3,Emily,22 4,Michael,40,Australia,Melbourne  Example Data - Success ID,Name,Age 2,Jane,35,California,USA 1,John,28 3,Emily,22 4,Michael,40,Australia,Melbourne 

Answer: Read it in as a single column by setting the separator to (hopefully) an unused utf8 character with no header and then use .str.split.list.to_struct followed by unnest to allow a dynamic number of columns. Then you have to rename the columns and slice out the first row. import polars as pl import io from warnings import catch_warnings, filterwarnings input_file = io.StringIO("""ID,Name,Age 1,John,28 2,Jane,35,California,USA 3,Emily,22 4,Michael,40,Australia,Melbourne""" ) input_file.seek(0) with catch_warnings(): filterwarnings("ignore") ## this suppresses the warning from `to_struct` which wants explicit field names. df = ( pl.read_csv(input_file, separator="\x00", has_header=False) .with_columns( pl.col("column_1") .str.split(",") .list.to_struct(n_field_strategy="max_width") ) .unnest("column_1") ) df = df.rename({x:y for x,y in zip(df.columns, df.row(0)) if y is not None}) df = df.slice(1,)  Now you've got a df of all strings. You could try to do a for loop with all the columns, trying to cast them but turns out that is slower (at least in a few tests that I did) than writing the existing df to a csv and then rereading it to force polars's auto-infer mechanism. from tempfile import NamedTemporaryFile with NamedTemporaryFile() as ff: df.write_csv(ff) ff.seek(0) df= pl.read_csv(ff)  If you've got enough memory then replacing the tempfile with an io.BytesIO() will be even faster.
Question: I'm using cartopy to draw a geomap. This is how I set up graticule rendering:  if graticule: gl = ax.gridlines( draw_labels=True, linewidth=0.8, color='gray', alpha=0.5, linestyle='--', x_inline=False, y_inline=True ) gl.xlocator = mticker.FixedLocator(np.arange(-180, 181, 10)) gl.ylocator = mticker.FixedLocator(np.arange(0, 91, 10)) gl.top_labels = False gl.bottom_labels = True gl.left_labels = True gl.right_labels = True gl.xlabel_style = {'size': 10, 'color': 'gray'} gl.ylabel_style = {'size': 10, 'color': 'gray'}  cartopy doesn't draw longitude labels to the left and to the right of the map. Only bottom and top (if on) labels are drawn:  With ax.set_extent([0, 40, 75, 85], crs=ccrs.PlateCarree()) added:  Latitude labels are okay being inline.

Answer: In draw_labels parameter, instead of setting it to True you can pass a dictionary with each 4 side and which coordinates you wish to see: gl = ax.gridlines( draw_labels={"bottom": "x", "left": "x", "right":"x"}, linewidth=0.8, color='gray', alpha=0.5, linestyle='--', x_inline=False, y_inline=True )  You will also need to remove these 4 lines: # gl.top_labels = False # gl.bottom_labels = True # gl.left_labels = True # gl.right_labels = True
Question: I am trying to plot split violin plots with Seaborn, i.e. a pair of KDE plots stacked against each other, typically to see the difference between distributions. My use case is very similar to the docs except I would like to superimpose custom box plots on top (as in this tutorial) However, I am having a strange alignment issue with the violin plots with respect to the X axis and I don't understand what I am doing differently from the docs... Here's code for a MRE with only split violins: import numpy as np import pandas as pd from matplotlib import pyplot as plt import seaborn as sns sns.set_theme() data1 = np.random.normal(0, 1, 1000) data2 = np.random.normal(1, 2, 1000) data = pd.concat( [ pd.DataFrame({"column": "1", "data1": data1, "data2": data2}), pd.DataFrame({"column": "2", "data3": data2, "data4": data1}), ], axis="rows", ) def mkplot(): fig, violin_ax = plt.subplots() sns.violinplot( data=data.melt(id_vars="column"), y="value", split=True, hue="variable", x="column", ax=violin_ax, palette="Paired", bw_method="silverman", inner=None, ) plt.show() mkplot()  This produces split violins whose middle is not aligned with the X axis label: mis-aligned violins (this is also true when "column" is of numeric type rather than str) It seems that box plots are also mis-aligned, but not with the same magnitude; you can use the function below with the same data def mkplot2(): fig, violin_ax = plt.subplots() sns.violinplot( data=data.melt(id_vars="column"), y="value", split=True, hue="variable", x="column", ax=violin_ax, palette="Paired", bw_method="silverman", inner=None, ) sns.boxplot( data=data.melt(id_vars="column"), y="value", hue="variable", x="column", ax=violin_ax, palette="Paired", width=0.3, flierprops={"marker": "o", "markersize": 3}, legend=False, dodge=True, ) plt.show() mkplot2()  mis-aligned violins+boxes How can I solve this ?

Answer: The issue is due to the NaNs that you have after melting. This makes 4 groups and thus the violins are shifted to account for those. You could plot the groups independently: data_flat = data.melt('column').dropna(subset='value') violin_ax = plt.subplot() pal = sns.color_palette('Paired') for i, (name, g) in enumerate(data_flat.groupby('column')): sns.violinplot( data=g, y='value', split=True, hue='variable', x='column', ax=violin_ax, palette=pal[2*i:2*i+2], bw_method='silverman', inner=None, )  Output:
Question: I'm trying to learn skopt, but I'm struggling to get even a simple multivariate minimization to run. import skopt def black_box_function(some_x, some_y): return -some_x + 2 - (some_y - 1) ** 2 + 1 BOUNDS = [(0, 100.0), (0, 100.0)] result = skopt.dummy_minimize(func=black_box_function, dimensions=BOUNDS)  When I run this, it seems to figure out that I want the search space for some_x to lie between 0 and 100, but it returns this error: TypeError: black_box_function() missing 1 required positional argument: 'some_y'.  How can I define the search space for both some_x and some_y?

Answer: Quoting the documentation  Function to minimize. Should take a single list of parameters and return the objective value.  So black_box_function should not have two parameters some_x, some_y, but a single parameter some_xy, that is a list of those two def black_box_function(some_xy): some_x, some_y = some_xy return -some_x + 2 - (some_y - 1) ** 2 + 1
Question: How to calculate horizontal median for numerical columns? df = pl.DataFrame({"ABC":["foo", "bar", "foo"], "A":[1,2,3], "B":[2,1,None], "C":[1,2,3]}) print(df) shape: (3, 4) ┌─────┬─────┬──────┬─────┐ │ ABC ┆ A ┆ B ┆ C │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪══════╪═════╡ │ foo ┆ 1 ┆ 2 ┆ 1 │ │ bar ┆ 2 ┆ 1 ┆ 2 │ │ foo ┆ 3 ┆ null ┆ 3 │ └─────┴─────┴──────┴─────┘  I want to achieve the same as with the below pl.mean_horizontal, but get median instead of the mean. I did not find existing expression for this. print(df.with_columns(pl.mean_horizontal(pl.col(pl.Int64)).alias("Horizontal Mean"))) shape: (3, 5) ┌─────┬─────┬──────┬─────┬─────────────────┐ │ ABC ┆ A ┆ B ┆ C ┆ Horizontal Mean │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ f64 │ ╞═════╪═════╪══════╪═════╪═════════════════╡ │ foo ┆ 1 ┆ 2 ┆ 1 ┆ 1.333333 │ │ bar ┆ 2 ┆ 1 ┆ 2 ┆ 1.666667 │ │ foo ┆ 3 ┆ null ┆ 3 ┆ 3.0 │ └─────┴─────┴──────┴─────┴─────────────────┘ 

Answer: There's no median_horizontal() at the moment, but you could use  pl.concat_list() to create list column out of all pl.Int64 columns. pl.Expr.list.median() to calculate median.  df.with_columns( pl.concat_list(pl.col(pl.Int64)).list.median().alias("Horizontal Median") )  shape: (3, 5) ┌─────┬─────┬──────┬─────┬───────────────────┐ │ ABC ┆ A ┆ B ┆ C ┆ Horizontal Median │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ f64 │ ╞═════╪═════╪══════╪═════╪═══════════════════╡ │ foo ┆ 1 ┆ 2 ┆ 1 ┆ 1.0 │ │ bar ┆ 2 ┆ 1 ┆ 2 ┆ 2.0 │ │ foo ┆ 3 ┆ null ┆ 3 ┆ 3.0 │ └─────┴─────┴──────┴─────┴───────────────────┘  Or you can use numpy integration (but this will probably be slower): import numpy as np df.with_columns( pl.Series("Horizontal Median", np.nanmedian(df.select(pl.col(pl.Int64)), axis=1)) )  shape: (3, 5) ┌─────┬─────┬──────┬─────┬───────────────────┐ │ ABC ┆ A ┆ B ┆ C ┆ Horizontal Median │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 ┆ f64 │ ╞═════╪═════╪══════╪═════╪═══════════════════╡ │ foo ┆ 1 ┆ 2 ┆ 1 ┆ 1.0 │ │ bar ┆ 2 ┆ 1 ┆ 2 ┆ 2.0 │ │ foo ┆ 3 ┆ null ┆ 3 ┆ 3.0 │ └─────┴─────┴──────┴─────┴───────────────────┘
Question: I have two dataframes with identical shape and want to update df1 with df2 if some conditions are met import pandas as pd from typing import Any df1 = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) print(df1, "\n") df2 = pd.DataFrame({"A": [7, 8, 9], "B": [10, 3, 12]}) print(df2, "\n") # Define a condition function def condition(x: Any) -> bool: """condition function to update only cells matching the conditions""" return True if x in [2, 7, 9] else False df1.update(df2) print(df1)  but if I use filter_func df1.update(df2,filter_func=condition) it fails with ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()  Unfortunately the docu is not very verbose. How to update a dataframe with conditions correctly?

Answer: Your function will receive a 1D numpy array (per column), it should be vectorized and return a boolean 1D array (callable(1d-array) -> bool 1d-array). Use numpy.isin to test membership: def condition(x): """condition function to update only cells matching the conditions""" return np.isin(x, [2, 7, 9]) df1.update(df2, filter_func=condition)  with a lambda: df1.update(df2, filter_func=lambda x: np.isin(x, [2, 7, 9]))  Alternatively, if you really can't vectorize with pure numpy functions (this should not be done here!), decorate with numpy.vectorize: @np.vectorize def condition(x: Any) -> bool: """condition function to update only cells matching the conditions""" return True if x in [2, 7, 9] else False df1.update(df2, filter_func=condition)  Updated df1:  A B 0 1 4 1 8 5 2 3 6
Question: I have a problem with a multi axis matplotlib plot. The code is close to what I want but somehow axes are getting mixed up. The ticks are missing on ax4 aka the green y-axis but only show up on ax2 (the red one) and the labels are duplicated and appear on both axes, ax2 and ax4. import numpy as np import matplotlib.pyplot as plt # Generate fake data distance = np.logspace(2, 4, num=50) a_detector = 10**4 / distance**1.5 b_detector = 10**5 / distance**1.6 c_detector = 10**3 / distance**1.4 d_detector = 10**2 / distance**1.2 # Create figure and axes fig, ax1 = plt.subplots(figsize=(20, 10)) ax1.plot(distance, a_detector, 'bo-', label='A') ax1.set_xlabel('Shower Plane Distance [meter]') ax1.set_ylabel('signal type I', color='blue') ax1.set_xscale('log') ax1.set_yscale('log') ax1.tick_params(axis='y', labelcolor='blue') ax2 = ax1.twinx() ax2.plot(distance, b_detector, 'ro-', label='B') ax2.set_ylabel('signal type II', color='red') ax2.set_yscale('log') ax2.tick_params(axis='y', labelcolor='red') ax3 = ax1.twinx() ax3.spines['right'].set_position(('outward', 90)) ax3.plot(distance, c_detector, 'ks-', label='C') ax3.set_ylabel('signal type III', color='black') ax3.set_yscale('log') ax3.tick_params(axis='y', labelcolor='black') ax4 = ax1.twinx() ax4.spines['left'].set_position(('outward', 90)) ax4.plot(distance, d_detector, 'g^-', label='D') ax4.set_ylabel('signal type IV', color='green') ax4.set_yscale('log') ax4.tick_params(axis='y', labelcolor='green') ax4.yaxis.set_label_position('left') ax4.yaxis.set_tick_params(labelleft=True) fig.legend(loc='upper right', bbox_to_anchor=(0.89, 0.86)) plt.show() 

Answer: You just need to replace: ax4.yaxis.set_tick_params(labelleft=True)  by: ax4.yaxis.set_tick_params(which='major', left=True, right=False, labelleft=True, labelright=False) ax4.yaxis.set_tick_params(which='minor', left=True, right=False, labelleft=True, labelright=False)  You put the yaxis major tick labels to the left but you also need to remove them from their original location with labelright=False. This has to be done also for the tick labels (previously it was the tick marks) and also for the minor ticks (default is major ticks).
Question: In bar.py foo is imported # bar.py from path import foo  In my current file bar is imported and I use the get_id function of foo: from path import bar bar.foo.get_id()   Mypy is complaining error: Module "bar" does not explicitly export attribute "foo" [attr-defined] 

Answer: Is it your own module? Use one of these two options that are suggested in the mypy docs; the __all__ is generally preferred # This will re-export it as bar and allow other modules to import it from foo import bar as bar # This will also re-export bar from foo import bar __all__ = ['bar']  If its a 3rd party; but also valid for your on modules Not all modules are designed and follow these suggestions you can disable/not check for the error, but first check:  Is is more valid to use foo directly? from path import foo foo.get_id()  I argue that the usage of module.non_child_module.function is often not justified. Also when writing your own modules it's more likely to create import-cycles when you put an unnecessary intermediate module in between.  If you think keeping bar.foo is more valid than using foo directly you can either add a # type: ignore[attr-defined]  Do not check for the error at all and use --no-implicit-reexport / implicit_reexport=False as many modules
Question: With a dataset with millions of records, I have items with various categories and measurements, and I'm trying to figure out how many of the records have changed, in particular when the category or measurement goes to NaN (or NULL from the database query) during the sequence. In SQL, I'd use some PARTITION style OLAP functions to do this, but seems like it should fairly straightforward in Python with Pandas, but I can't quite wrap my head around the vectorized notation. I've tried various df.groupby clauses and lambda functions but nothing quite gets it in the required format - basically, the df.groupby('item')['measure'] in this example, the first row of the grouped subset of item & measure always returns True, where I'd like to it to be False or NaN. Simply put, they are false positives. I understand from pandas' perspective, it's a change since the first x.shift() would be NaN, but I can't figure out how to filter that or handle it in the lambda function. Sample Code: import pandas as pd import numpy as np test_df = pd.DataFrame({'item': [20, 20, 20, 20, 20, 20, 20, 20, 30, 30, 30, 30, 30, 30, 30, 30, 40, 40, 40, 40, 40, 40, 40, 40 ], 'measure': [1, 1, 1, 3, 3, 3, 3, 3, 6, 6, 6, 6, 6, 7, 7, 7, 10, 10, 10, 10, 10, 10, 10, 10 ], 'cat': ['a', 'a', 'a', 'b', 'b', 'b', 'b', 'b', 'c', 'c', 'c', 'c', 'c', 'd', 'd', 'd', 'e', 'e', 'e', 'e', 'e', 'e', 'e', 'e']}) test_df['measure_change'] = test_df.groupby('item')['measure'].transform(lambda x: x.shift() != x) test_df['cat_change'] = test_df.groupby('item')['cat'].transform(lambda x: x.shift() != x)  In the output below, as an example, rows 0, 8, and 16, the measure_change should be False. So all of item 40 would have measure_change == False and that would indicate no changes with that item. Any & all suggestions are appreciated. (cat_change set up the same way)    # item measure measure_change     0 20 1 True   1 20 1 False   2 20 1 False   3 20 3 True   4 20 3 False   5 20 3 False   6 20 3 False   7 20 3 False   8 30 6 True   9 30 6 False   10 30 6 False   11 30 6 False   12 30 6 False   13 30 7 True   14 30 7 False   15 30 7 False   16 40 10 True   17 40 10 False   18 40 10 False   19 40 10 False   20 40 10 False   21 40 10 False   22 40 10 False   23 40 10 False   

Answer: You can use a combination of groupby.diff and fillna to achieve this. We compare the row difference with 0 to find any rows where measure changed: test_df['measure_change'] = test_df.groupby('item')['measure'].diff().fillna(0) != 0  Result:  item measure measure_change 0 20 1 False 1 20 1 False 2 20 1 False 3 20 3 True 4 20 3 False 5 20 3 False 6 20 3 False 7 20 3 False 8 30 6 False 9 30 6 False 10 30 6 False 11 30 6 False 12 30 6 False 13 30 7 True 14 30 7 False 15 30 7 False 16 40 10 False 17 40 10 False 18 40 10 False 19 40 10 False 20 40 10 False 21 40 10 False 22 40 10 False 23 40 10 False   Alternativly, if you have strings to compare as well you can add a secondary condition checking the shift value for nans: x.shift().notna(). test_df['measure_change'] = test_df.groupby('item')['measure'].transform(lambda x: (x != x.shift()) & (x.shift().notna()))
Question: I have an image, from which I would like to filter some colors: if a given pixel is in my set of colors, I would like to replace it by a white pixel. With an image called original of shape (height, width, 3) and a color like ar=np.array([117,30,41]), the following works fine: mask = (original == ar).all(axis=2) original[mask] = [255, 255, 255]  The trouble is, the set of colors I want to exclude is quite big (~37000). Using the previous code in a loop (for ar in colors) works again, but is quite slow. Is there a faster way to do it? For now, my set of colors is in a numpy array of shape (37000, 3). I'm sure that all of these colors are present on my image, and I'm also sure that they are unique.

Answer: A simple way to solve this would be a look up table. A look up table with a boolean for every color would only cost 256 * 256 * 256 * 1 bytes = 16 MiB, and would enable you to determine if a color is in your list of disallowed colors in constant time. Here is an example. This code generates an image with multiple colors. It filters out some of those colors using two approaches. The first approach is the one you describe in the question. The second approach is the lookup table. import numpy as np # Only used for generating image. Skip this if you already have an image. image_colors = np.array([ (100, 100, 100), (200, 200, 200), (255, 255, 0), (255, 0, 0), ]) image_colors_to_remove = [ (255, 255, 0), (255, 0, 0), ] # Generate image resolution = (800, 600) np.random.seed(42) image = np.random.randint(0, len(image_colors), size=resolution) image = np.array(image_colors)[image].astype(np.uint8) # image = np.random.randint(0, 256, size=(*resolution, 3)) # Slow approach def remove_colors_with_for(image, image_colors_to_remove): image = image.copy() for c in image_colors_to_remove: mask = (image == c).all(axis=2) image[mask] = [255, 255, 255] return image # Fast approach def remove_colors_with_lookup(image, image_colors_to_remove): image = image.copy() colors_remove_lookup = np.zeros((256, 256, 256), dtype=bool) image_colors_to_remove = np.array(image_colors_to_remove).T colors_remove_lookup[tuple(image_colors_to_remove)] = 1 image_channel_first = image.transpose(2, 0, 1) mask = colors_remove_lookup[tuple(image_channel_first)] image[mask] = [255, 255, 255] return image new_image = remove_colors_with_for(image, image_colors_to_remove) new_image2 = remove_colors_with_lookup(image, image_colors_to_remove) print("Same as for loop?", np.all(new_image2 == new_image))