eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a csv file with a partial format of something like: field1,bmm/bdd/byyyy,emm/edd/eyyyy,field4....  I am successfully creating a json file like this: { "field1": [ { "begDate": byyyybmmbdd, "endDate": eyyyyemmedd, "score": field4, .....  The python script that I was using works fine but it gives me deprecation warning: from datetime import datetime dateparse = lambda x: datetime.strptime(x, '%m/%d/%Y') df = pd.read_csv("input.csv", parse_dates=['Start', 'End'], date_parser=dateparse) df['Start'] = df['Start'].astype(str) df['End'] = df['End'].astype(str) df['score'] = df['score'].round(decimals=3) res = {} for a1, df_gp in df.groupby('field1'): res[a1] = df_gp.drop(columns='field1').to_dict(orient='records') print(json.dumps(res, indent=4).lower())   FutureWarning: The argument 'date_parser' is deprecated and will be removed in a future version. Please use 'date_format' instead, or read your data in as 'object' dtype and then call 'to_datetime'.  Id like to be able to run the script w/o the warning so I modified the script accordingly: dateparse = lambda x: datetime.strptime(x, '%m/%d/%Y') df = pd.read_csv("input.csv", parse_dates=['Start', 'End'], date_format=dateparse)  I also tried this: dateparse = lambda x: datetime.strptime(x, '%m/%d/%Y').strftime("%Y%m%d") df = pd.read_csv("input.csv", parse_dates=['Start', 'End'], date_format=dateparse)  but the json output gives me the wrong date format: { "field1": [ { "begDate": bmm/bdd/byyyy, "endDate": emm/edd/eyyyy, "score": 0.0, ....  Are there any suggestions on how to get around this Warning message while receiving the desired output?

Answer: You can avoid the deprecation warning by not trying to replace the deprecated date_parser with a callable in date_format (which expects a string, not a function). Instead, load the dates as objects and then convert them with pd.to_datetime and dt.strftime to get the format you want. For example: import pandas as pd from datetime import datetime import json # Read the CSV without a custom parser (dates will be parsed based on 'parse_dates') df = pd.read_csv("input.csv", parse_dates=['Start', 'End']) # Now convert the date columns to the desired format (e.g. "YYYYMMDD") df['Start'] = pd.to_datetime(df['Start'], format='%m/%d/%Y').dt.strftime('%Y%m%d') df['End'] = pd.to_datetime(df['End'], format='%m/%d/%Y').dt.strftime('%Y%m%d') df['score'] = df['score'].round(3) # Group and convert to the desired JSON structure res = {} for a1, df_gp in df.groupby('field1'): res[a1] = df_gp.drop(columns='field1').to_dict(orient='records') print(json.dumps(res, indent=4).lower())  This way, you get rid of the warning and still achieve your desired JSON output.
Question: I want to insert a column at a specific index in a Pandas DataFrame, but only assign values to certain rows. Currently, I am doing it in two steps: df = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 20, 30, 40, 50] }) df.insert(1, 'NewCol', None) df.loc[[1, 3], 'NewCol'] = ['X', 'Y']  Is there a more concise way to achieve this in a single operation?

Answer: Provide a Series with the correct indices to insert: df.insert(1, 'NewCol', pd.Series(['X', 'Y'], index=[1, 3]))  Output:  A NewCol B 0 1 NaN 10 1 2 X 20 2 3 NaN 30 3 4 Y 40 4 5 NaN 50
Question: I was looking through some codes and saw this line, numpy.random.normal(size=x.shape). Where, x = numpy.linspace(1, 2, 100). I don't understand what this does. I've only come across, np.random.normal(size=1) before. Can someone please explain the difference in both the cases and their use.

Answer: From numpy.random.normal  size: int or tuple of ints, optional Output shape. If the given shape is, e.g., (m, n, k), then m * n * k samples are drawn. If size is None (default), a single value is returned if loc and scale are both scalars. Otherwise, np.broadcast(loc, scale).size samples are drawn.  shape return a tuple. If you send it to the size parameter the size will be multiplication of the values (the result array will have the same shape) arr = numpy.array([1, 2, 3]) print(arr.shape) random_arr = numpy.random.normal(size=arr.shape) print(random_arr) # Output # (3,) # [ 0.02756549 -0.52115646 -2.32361849] arr = numpy.array([[1, 2, 3], [4, 5, 6]]) print(arr.shape) random_arr = numpy.random.normal(size=arr.shape) print(random_arr) # Output # (2, 3) # [[ 1.10564417 0.32478606 -1.71487667] # [ 0.5461406 0.51505975 0.2158163 ]] arr = numpy.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) print(arr.shape) random_arr = numpy.random.normal(size=arr.shape) print(random_arr) # Output # (3, 3) # [[ 0.75194289 -1.1696558 1.05238044] # [-0.68043824 0.98258701 3.48030454] # [-0.84775259 -0.49676403 1.33367409]]
Question: I'm trying to create the "Related Quantities" column of a dataframe given the existing "Item", "Quantity", and "Related Items" columns.     Item Quantity Related Items Related Quantities     0 Flowers 1 ['Bushes'] {'Bushes': 2}   1 Bushes 2 ['Flowers'] {'Flowers': 1}   2 Cars 3 ['Trucks', 'Motorcycles'] {'Trucks': 4, 'Motorcycles': 5}   3 Trucks 4 ['Cars', 'Motorcycles'] {'Cars': 3, 'Motorcycles': 5}   4 Motorcycles 5 ['Cars', 'Trucks'] {'Cars': 3, 'Trucks': 4}    The values in the dictionaries will be used as the input of a function to generate another column later  I believe I have a line that will make the dictionary for a single row, which I could use to fill out the column using iterrows() and a for loop: dictionary = {item : df.loc[df['Item'] == item,['Quantity']][idx] for item in related_items_list}  (where idx is something to grab the corresponding index of the one row left after filtering by 'Item', and related_items_list is the value grabbed from the 'Related Items' column of the current row in the loop) But I'm trying to make something with df.apply() instead, in hopes that it will be more performant. Is there some way to allow the function called in apply() to access the whole dataframe instead of just the row passed to it? I think I may be way overcomplicating this. Is trying to use apply() for performance a waste of time? Should I just be using a loop to feed the 'Quantity' data into the function directly instead of making this column at all? I worry that will also hurt performance EDIT Thank you. It looks like to_dict() is somewhat faster than making the lookup dict using a for loop. number of rows in df: 704400 Testing with 100 iterations: time of 'for item,qty in zip of two columns' method: 19.455784299999998 time of 'df.set_index(col1)[col2].to_dict()' method: 11.409341199999997 

Answer: Using df.apply with a lookup dictionary. import pandas as pd data = {'Item': ['Flowers', 'Bushes', 'Cars', 'Trucks', 'Motorcycles'], 'Quantity': [1, 2, 3, 4, 5], 'Related Items': [['Bushes'], ['Flowers'], ['Trucks', 'Motorcycles'], ['Cars', 'Motorcycles'], ['Cars', 'Trucks']]} df = pd.DataFrame(data) # Creates a dictionary for fast quantity lookups item_quantities = df.set_index('Item')['Quantity'].to_dict() def create_related_quantities(row): related_quantities = {} for item in row['Related Items']: quantity = item_quantities.get(item) # Get quantity from lookup dictionary or None if not found if quantity is not None: # Only add to dict if quantity exists related_quantities[item] = quantity return related_quantities # Applies the create_related_quantities function to each row (axis=1) of the DataFrame df['Related Quantities'] = df.apply(create_related_quantities, axis=1) print(df)  The new column created  Item ... Related Quantities 0 Flowers ... {'Bushes': 2} 1 Bushes ... {'Flowers': 1} 2 Cars ... {'Trucks': 4, 'Motorcycles': 5} 3 Trucks ... {'Cars': 3, 'Motorcycles': 5} 4 Motorcycles ... {'Cars': 3, 'Trucks': 4}
Question: I have a django project where I have some database tables. One of the database tables is designed to store messages and their titles. This helps me to create/alter these messages from my django-admin. Now I want to initialize a variable (as a dictionary) from this table as follows : MY_MSGS = {record.name : {'title':record.title, 'message':record.message} for record in MyTable.objects.all()}  This must happen at the server startup for now. MY_MSGS must be accessible to the different view-files. Later I would want to periodically update MY_MSGS by reading MyTable again. So I want that My_MSGS behaves as a all global to all my view-files and should be initialized after the startup is complete. FYI I have multiple view-files that are all imported from views.py. Also this is a very small table with just about maximum 15 messages and so I do not mind holding this data in memory

Answer: I think the main concern is that you should not run the query immediately, but after Django has initialized the models, etc. We can do that by postponing the load procedure, and do it when we really need a message, with: def get_message(name): cache = get_message.cache if cache is None: cache = get_message.cache = { record.name: {'title': record.title, 'message': record.message} for record in MyTable.objects.all() } return cache.get(name) get_message.cache = None and thus use this as a function, like: get_message(my_record_name) then it is of course still important to make sure you never call the function during initialization, so not pass this as a default=… [Django-doc], etc. of a method field for example. An extra advantage is that as long as you don't need any message, you don't fetch these. If we do, we will not do it a second time. But usually that might be a problem: typically you don't restart the Django server very often in production, so the messages can remain the old ones for months.
Question: I have a dataframe with 6 columns: 'Name', 'A', 'B', 'C', 'Val', 'Category' It looks like this: Name A B C Val Category x 1.1 0 0.2 NA NA y 0 0.1 0 NA NA z 0.5 0.1 0.3 NA NA  I want to expand the dataframe such that for each value that is not 0 in columns 'A', 'B', 'C' you get an extra row. The column 'Val' is assigned the non-zero value that led to the expansion and the 'Category' is arbitrarily based on where the value came from. The result should look like this: Name A B C Val Category x 1.1 0 0.2 1.1 first x 1.1 0 0.2 0.2 third y 0 0.1 0 0.1 second z 0.5 0.1 0.3 0.5 fisrt z 0.5 0.1 0.3 0.1 second z 0.5 0.1 0.3 0.3 third  This is probably the wrong approach, but I thought since I only have three columns I should be repeating all the rows 3 times by using the repeat function on the index and then looping through the rows based on a for loop with a skip to apply 3 functions to assign the target and AICN all rows and then dropping rows where the target is 0. def targeta(row): target = row val = 'first' return target, val def targetb(row): target = row val = 'second' return target, val def targetc(row): target = row val = 'third' return target, val df_repeat = df.loc[df.index.repeat(3)] for i in range(1,len(df_repeat)-3,3): df_repeat.iloc[i][['Target','Category']]=targeta(df_repeat.iloc[i]['A']) df_repeat.iloc[i+1][['Target','Category']]=targetb(df_repeat.iloc[i+1]['B']) df_repeat.iloc[i+2][['Target','Category']]=targetc(df_repeat.iloc[i+2]['C'])  I only got to this point and realized I am getting an empty dataframe. Any suggestions on what to do?

Answer: You could replace the 0s with NaNs, rename the columns to your categories, reshape to long with stack, and join back to the original to duplicate the rows: out = (df .drop(columns=['Val', 'Category']) .join(df[['A', 'B', 'C']] .set_axis(['first', 'second', 'third'], axis=1) .rename_axis(columns='Category') .replace(0, pd.NA) .stack() .rename('Val') .reset_index(-1) ) )  Output:  Name A B C Category Val 0 x 1.1 0.0 0.2 first 1.1 0 x 1.1 0.0 0.2 third 0.2 1 y 0.0 0.1 0.0 second 0.1 2 z 0.5 0.1 0.3 first 0.5 2 z 0.5 0.1 0.3 second 0.1 2 z 0.5 0.1 0.3 third 0.3
Question: The code: import sys import numpy as np print(f"We are using Python {sys.version}", file=sys.stderr) print(f"We are using numpy version {np.__version__}", file=sys.stderr) # 2.2.1 def find_non_numpy_floats(x: any) -> bool: if not (isinstance(x, np.float64)): print(f"Found non-numpy.float64: {x} of type {type(x)}", file=sys.stderr) return False else: return True w: np.ndarray = np.zeros((2, 2), dtype=np.float64) np.vectorize(lambda x: find_non_numpy_floats(x))(w) assert (np.all(np.vectorize(lambda x: isinstance(x, np.float64))(w))), "try to keep using the numpy floats"  I'm expecting Numpy.zeros to generate an array of Numpy float64, which are not the same as Python float if I understand correctly (IEEE 64-bit floats vs something Python specific?) However the above results in: We are using Python 3.13.1 (main, Dec 9 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] We are using numpy version 2.2.1 Found non-numpy.float64: 0.0 of type <class 'float'> Found non-numpy.float64: 0.0 of type <class 'float'> Found non-numpy.float64: 0.0 of type <class 'float'> Found non-numpy.float64: 0.0 of type <class 'float'>  and an assertion error. Why is that and how can I fix this (and should I want to?)

Answer: numpy.vectorize converts the array to an array of object dtype first:  # Convert args to object arrays first inputs = [asanyarray(a, dtype=object) for a in args]  I don't know why. I can think of a few plausible reasons, but nothing stands out as a clear motivator. In any case, converting to object dtype builds an array of ordinary Python scalar objects, rather than an array of NumPy scalars.  It is not possible for an array of float64 dtype to contain ordinary Python floats. An array of float64 dtype has a buffer of raw 8-byte floating-point values, not Python objects. They're not even instances of numpy.float64 - NumPy has to construct numpy.float64 wrapper objects on access if you try to access an individual element.
Question: Some Polars operations, such as .sort() fail when passed a column with a Nested Type. This (sensible) choice about sort means I cannot use my usual sorting pattern of df.sort(pl.all()). import polars as pl NESTED_TYPES = [ pl.List, pl.Array, pl.Object, pl.Struct ] pl.exclude(NESTED_TYPES)  Result: *.exclude([Dtype(List(Null)), Dtype(Array(Null, 0)), Dtype(Object("object", None)), Dtype(Struct([]))])  Is there a way to select (or exclude) only nested types? The Selectors Documentation has many ideas but nothing seems right for this.

Answer: It is not very well supported yet, see https://github.com/pola-rs/polars/issues/9971 As a workaround, you can specify "all types except non-nested types" using the code shown in that Issue, or just loop over all dtypes you care about and specify pl.List(dtype) for each of them, but there aren't any good ways to say "all list dtypes" yet.
Question: What is the difference between ConfigDict and dict? ConfigDict: TypeAlias = dict[str, Union[str, list[str]]]  What are the advantages of using ConfigDict? https://github.com/pytest-dev/pytest/pull/13193/files#diff-f1d27932fbd9530086080aa8df367309881fe90f204cdd69102ba59758644761

Answer: ConfigDict in the PR you linked to is a type alias to a dict where the key is a string and the value is either a string or a list of strings. In runtime, there isn't any functional difference. It's mainly useful in development time to save having to type the long dict definition each time and to avoid mistakenly using a different definition (e.g., using dict(str, str) by mistake.
Question: In uv, I want to add a Git repository as a dependency which is easy in itself, but if we commit in the Git repository of the dependency, I want uv sync to do a Git pull for this Git repository. Basically, do a Git pull on the dependency Git repository and then apply the changed code for the dependency to my current virtual env. Here is what I tried: uv add git+https://github.com/PowerInsight/quantstats.git  Then I make a commit on https://github.com/PowerInsight/quantstats.git in a Python file. Then I run this in the repository that references https://github.com/PowerInsight/quantstats.git uv sync  The file I changed in the Git repository never gets updated in my .venv folder for the referenced dependency: .venv\Lib\site-packages\quantstats Then tried the same thing with a specific branch: uv add git+https://github.com/PowerInsight/quantstats.git --branch main  It is the same problem; it does not get updated on commit. Then I tried adding this to both the pyproject.toml of my main project and the dependency Git repository: [tool.uv] cache-keys = [{ git = { commit = true } }]  I also tried setting this package to always get reinstalled: [tool.uv] reinstall-package = ["quantstats"]  What do I need to do for uv to pull from the Git repository dependency on any commit?

Answer: TLDR. You can use uv sync --upgrade  Explanation. Per documentation of uv sync:  Syncing ensures that all project dependencies are installed and up-to-date with the lockfile.  The lockfile doesn't change when the remote repository is updated, but it can be upgraded using uv lock --upgrade  or uv sync --upgrade  allowing for package upgrades.
Question: I'm trying to use polars dataframe where I would like to select the first and last row per group. Here is a simple example selecting the first row per group: import polars as pl df = pl.DataFrame( { "a": [1, 2, 2, 3, 4, 5], "b": [0.5, 0.5, 4, 10, 14, 13], "c": [True, True, True, False, False, True], "d": ["Apple", "Apple", "Apple", "Banana", "Banana", "Banana"], } ) result = df.group_by("d", maintain_order=True).first() print(result)  Output: shape: (2, 4) ┌────────┬─────┬──────┬───────┐ │ d ┆ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 ┆ bool │ ╞════════╪═════╪══════╪═══════╡ │ Apple ┆ 1 ┆ 0.5 ┆ true │ │ Banana ┆ 3 ┆ 10.0 ┆ false │ └────────┴─────┴──────┴───────┘  This works good and we can use .last to do it for the last row. But how can we combine these in one group_by?

Answer: As columns You could use agg, you will have to add a suffix (or prefix) to differentiate the columns names: result = (df.group_by('d', maintain_order=True) .agg(pl.all().first().name.suffix('_first'), pl.all().last().name.suffix('_last')) )  Output: ┌────────┬─────────┬─────────┬─────────┬────────┬────────┬────────┐ │ d ┆ a_first ┆ b_first ┆ c_first ┆ a_last ┆ b_last ┆ c_last │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │ ╞════════╪═════════╪═════════╪═════════╪════════╪════════╪════════╡ │ Apple ┆ 1 ┆ 0.5 ┆ true ┆ 2 ┆ 4.0 ┆ true │ │ Banana ┆ 3 ┆ 10.0 ┆ false ┆ 5 ┆ 13.0 ┆ true │ └────────┴─────────┴─────────┴─────────┴────────┴────────┴────────┘  As rows If you want multiple rows, then you would need to concat: g = df.group_by('d', maintain_order=True) result = pl.concat([g.first(), g.last()]).sort(by='d', maintain_order=True)  Output: ┌────────┬─────┬──────┬───────┐ │ d ┆ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 ┆ bool │ ╞════════╪═════╪══════╪═══════╡ │ Apple ┆ 1 ┆ 0.5 ┆ true │ │ Apple ┆ 2 ┆ 4.0 ┆ true │ │ Banana ┆ 3 ┆ 10.0 ┆ false │ │ Banana ┆ 5 ┆ 13.0 ┆ true │ └────────┴─────┴──────┴───────┘  Or using filter with int_range+over: result = df.filter((pl.int_range(pl.len()).over('d') == 0) |(pl.int_range(pl.len(), 0, -1).over('d') == 1) )  Output: ┌─────┬──────┬───────┬────────┐ │ a ┆ b ┆ c ┆ d │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ bool ┆ str │ ╞═════╪══════╪═══════╪════════╡ │ 1 ┆ 0.5 ┆ true ┆ Apple │ │ 2 ┆ 4.0 ┆ true ┆ Apple │ │ 3 ┆ 10.0 ┆ false ┆ Banana │ │ 5 ┆ 13.0 ┆ true ┆ Banana │ └─────┴──────┴───────┴────────┘
Question: In scipy, the function scipy.interpolate.make_interp_spline() can be batched since its x argument must be one-dimensional with shape (m,) and its y argument can have shape (m, ...). However, the function scipy.interpolate.make_smoothing_spline() only accepts a y argument of shape (m,). Is there a simple way to batch the behavior of make_smoothing_spline() so it has the same behavior as make_interp_spline()? I was thinking of using numpy.vectorize(), but here I'm not batching operations on an array, I need a single function as output. I guess I could just implement a loop and make a nested list of splines, but I was wondering if there would be a neater way. Probably some combination of decorators but I'm twisting my brain in knots... EDIT: Developers seem to be aware of this issue here.

Answer: The PR that added batch support to make_smoothing_spline happened to be merged a few hours before this post. https://github.com/scipy/scipy/pull/22484 The feature will be available in SciPy 1.16, or you can get it early in the next nightly wheels. https://anaconda.org/scientific-python-nightly-wheels/scipy See also the BatchSpline class used in the tests of that PR. class BatchSpline: # BSpline-like class with reference batch behavior def __init__(self, x, y, axis, *, spline, **kwargs): y = np.moveaxis(y, axis, -1) self._batch_shape = y.shape[:-1] self._splines = [spline(x, yi, **kwargs) for yi in y.reshape(-1, y.shape[-1])] self._axis = axis def __call__(self, x): y = [spline(x) for spline in self._splines] y = np.reshape(y, self._batch_shape + x.shape) return np.moveaxis(y, -1, self._axis) if x.shape else y def integrate(self, a, b, extrapolate=None): y = [spline.integrate(a, b, extrapolate) for spline in self._splines] return np.reshape(y, self._batch_shape) def derivative(self, nu): res = copy.deepcopy(self) res._splines = [spline.derivative(nu) for spline in res._splines] return res def antiderivative(self, nu): res = copy.deepcopy(self) res._splines = [spline.antiderivative(nu) for spline in res._splines] return res