eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Polars dataframes have an is_empty attribute: import polars as pl df = pl.DataFrame() df.is_empty() # True df = pl.DataFrame({"a": [], "b": [], "c": []}) df.is_empty() # True  This is not the case for Polars lazyframes, so I devised the following helper function: def is_empty(data: pl.LazyFrame) -> bool: return ( data.width == 0 # No columns or data.null_count().collect().sum_horizontal()[0] == 0 # Columns exist, but are empty ) other = pl.LazyFrame() other.pipe(is_empty) # True other = pl.LazyFrame({"a": [], "b": [], "c": []}) other.pipe(is_empty) # True  Is there a better way to do this? By better, I mean either without collecting or less memory-intensive if collecting can not be avoided.

Answer: As explained in the comments, "A LazyFrame doesn't have length. It is a promise on future computations. If we would do those computations implicitly, we would trigger a lot of work silently. IMO when the length is needed, you should materialize into a DataFrame and cache that DataFrame so that that work isn't done twice". So, calling collect is inevitable, but one can limit the cost by collecting only the first row (if any) with Polars limit, as suggested by @Timeless: import polars as pl df = pl.LazyFrame() df.limit(1).collect().is_empty() # True df= pl.LazyFrame({"a": [], "b": [], "c": []}) df.limit(1).collect().is_empty() # True df = pl.LazyFrame({col: range(100_000_000) for col in ("a", "b", "c")}) df.limit(1).collect().is_empty() # False, no memory cost
Question: I want to replace the categories with their occurrence frequency. My dataframe is lazy and currently I cannot do it without 2 passes over the entire data and then one pass over a column to get the length of the dataframe. Here is how I am doing it: Input: df = pl.DataFrame({"a": [1, 8, 3], "b": [4, 5, None], "c": ["foo", "bar", "bar"]}).lazy() print(df.collect()) output: shape: (3, 3) ┌─────┬──────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═════╪══════╪═════╡ │ 1 ┆ 4 ┆ foo │ │ 8 ┆ 5 ┆ bar │ │ 3 ┆ null ┆ bar │ └─────┴──────┴─────┘  Required output: shape: (3, 3) ┌─────┬──────┬────────────────────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═════╪══════╪════════════════════╡ │ 1 ┆ 4 ┆ 0.3333333333333333 │ │ 8 ┆ 5 ┆ 0.6666666666666666 │ │ 3 ┆ null ┆ 0.6666666666666666 │ └─────┴──────┴────────────────────┘  transformation code: l = df.select("c").collect().shape[0] rep = df.group_by("c").len().collect().with_columns(pl.col("len")/l).lazy() df_out = df.with_context(rep.select(pl.all().name.prefix("context_"))).with_columns(pl.col("c").replace(pl.col("context_c"), pl.col("context_len"))).collect() print(df_out) output: shape: (3, 3) ┌─────┬──────┬────────────────────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═════╪══════╪════════════════════╡ │ 1 ┆ 4 ┆ 0.3333333333333333 │ │ 8 ┆ 5 ┆ 0.6666666666666666 │ │ 3 ┆ null ┆ 0.6666666666666666 │ └─────┴──────┴────────────────────┘  As you can see I am collecting the data 2 times full and there is one collect over a single column. Can I do better?

Answer: pl.len() will evaluate to the "column length". You can also use it in a group context (agg/over) as a way to count the values. df.with_columns(pl.len().over("c") / pl.len()).collect()  shape: (3, 3) ┌─────┬──────┬──────────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ f64 │ ╞═════╪══════╪══════════╡ │ 1 ┆ 4 ┆ 0.333333 │ │ 8 ┆ 5 ┆ 0.666667 │ │ 3 ┆ null ┆ 0.666667 │ └─────┴──────┴──────────┘  By grouping by the values, their "frequency count" is the group length. >>> df.group_by("c").len() shape: (2, 2) ┌─────┬─────┐ │ c ┆ len │ │ --- ┆ --- │ │ cat ┆ u32 │ ╞═════╪═════╡ │ foo ┆ 1 │ │ bar ┆ 2 │ └─────┴─────┘
Question: I'm parsing text in Python using regex that typically looks like some version of this: JT Meta Platforms, Inc. - Class A Common Stock (META) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 F S: New S O: Morgan Stanley - Select UMA Account # 1 JT Microsoft Corporation - Common Stock (MSFT) [ST]S (partial) 02/08/2024 03/05/2024 $1,001 - $15,000 F S: New S O: Morgan Stanley - Select UMA Account # 1 JT Microsoft Corporation - Common Stock (MSFT) [OP]P 02/13/2024 03/05/2024 $500,001 - $1,000,000 F S: New S O: Morgan Stanley - Portfolio Management Active Assets Account D: Call options; Strike price $170; Expires 01/17 /2025 C: Ref: 044Q34N6  I've set up a regex to extract 'ticker' (eg, MSFT, META) that looks like this: r"\(([A-Z]+\.?[A-Z]*?)\)"  This pulls capitalized adjacent chars (eg, IBM, T) sitting within parens, and also allows there to be an optional period (happens occasionally eg, BRK.B) for certain situations. The below example is matching starting with the '(BM)' characters, which are included in a footnote for a previous transaction, and not valid tickers. The logic to identify these would be that they're preceded by the 'F S:' footnote designation, where there can be slight deviations in spaces but will include always include 'FS:'. How to exclude these errant situations using regex in Python? Alibaba Group Holding Limited American Depositary Shares each representing eight Ordinary share (BABA) [ST]S 01/19/2024 01/22/2024 $1,001 - $15,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA IRA (BM) Alphabet Inc. - Class C Capital Stock (GOOG) [ST]S 01/19/2024 01/22/2024 $1,001 - $15,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA ROTH IRA (JM) Alphabet Inc. - Class C Capital Stock (GOOG) [ST]S 01/19/2024 01/22/2024 $15,001 - $50,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA IRA (BM) Amazon.com, Inc. (AMZN) [ST] S 01/19/2024 01/22/2024 $15,001 - $50,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA IRA (BM) Amazon.com, Inc. (AMZN) [ST] S 01/19/2024 01/22/2024 $1,001 - $15,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA ROTH IRA (JM) Amazon.com, Inc. (AMZN) [ST] S 01/19/2024 01/22/2024 $1,001 - $15,000 F S: New S O: Iron Gate GA Brokerage Portfolio > IGGA ROTH IRA (BM)Filing ID #20024354 

Answer: Since you expect only single matches per line, you can use ^(?![A-Z] [A-Z]:).*\(([A-Z]+\.?[A-Z]*)\)  See the regex demo. Details:  ^ - start of string (?![A-Z] [A-Z]:) - exclude the string that starts with an uppercase letter + space + uppercase letter + : .* - any zero or more chars other than line break chars as many as possible \( - a ( char ([A-Z]+\.?[A-Z]*) - Group 1: one or more uppercase letters, an optional dot and then zero or more uppercase letters \) - a ) char.
Question: I have a C++/Qt application where I am running a Python interpreter session within the process of the main application. I have built a 'python console' as a QPlainTextEdit widget, which handles the input and output for the interpreter using the Python C API. The point of all this is that Python will have direct access to data that I'm holding inside the main application. All is well, except that I want to be able to end the current interpreter session and restart it without exiting the main application. Currently, I'm trying the obvious way: Py_Initialize(); // Run the main session... Py_FinalizeEx(); // Restart the session Py_Initialize();  From reading other posts and Python's documentation, there are possibly issues with reloading certain modules after finalizing the interpreter due to memory leaks. This seems to be true in my case: re-importing certain modules (like 'numpy') but not others (like 'sys') will trigger an exception and fail. Is there a good workaround strategy to restart the interpreter without these issues? For instance, if I spawn a sub-interpreter, will finalizing it and restarting a new sub-interpreter run into the same problems I'm having? The alternative I was trying to avoid is to run Python out-of-process, which I think would allow me to restart by basically killing the process and starting a new one. Any strategy advice would be much appreciated.

Answer: From Py_FinalizeEx documentation:  Bugs and caveats: The destruction of modules and objects in modules is done in random order; this may cause destructors (del() methods) to fail when they depend on other objects (even functions) or modules. Dynamically loaded extension modules loaded by Python are not unloaded. Small amounts of memory allocated by the Python interpreter may not be freed (if you find a leak, please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.  Specifically note that:  Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.  (emphasys is mine) It is clear that the design does not support finalization and reinitialization as you attempt. It is therefore recommended that you initialize the python engine once (when your process starts or at the first time you need it), and only finalize it upon exit (or at least when you are sure you will not need it anymore). Even if you find some workaround that will work in a specific scenario I do not think it is a good idea given the information above. Note: If you have some requirement that mandates creating a new Python envirionment for each session or operation, the best approach is to run Python as an external process.
Question: I have the following dictionary that contains node-neighbor-weight pairs: graph = { "A": {"B": 3, "C": 3}, "B": {"A": 3, "D": 3.5, "E": 2.8}, "C": {"A": 3, "E": 2.8, "F": 3.5}, "D": {"B": 3.5, "E": 3.1, "G": 10}, "E": {"B": 2.8, "C": 2.8, "D": 3.1, "G": 7}, "F": {"G": 2.5, "C": 3.5}, "G": {"F": 2.5, "E": 7, "D": 10}, }  The visual representation of the graph is this:  How do I go from that dictionary to a networkx graph object: import networkx as nx G = nx.Graph() # YOUR SUGGESTION HERE # 

Answer: You need to add the dictionary as edges to the graph import networkx as nx from matplotlib import pyplot as plt graph = { "A": {"B": 3, "C": 3}, "B": {"A": 3, "D": 3.5, "E": 2.8}, "C": {"A": 3, "E": 2.8, "F": 3.5}, "D": {"B": 3.5, "E": 3.1, "G": 10}, "E": {"B": 2.8, "C": 2.8, "D": 3.1, "G": 7}, "F": {"G": 2.5, "C": 3.5}, "G": {"F": 2.5, "E": 7, "D": 10}, } G = nx.Graph() for node, neighbors in graph.items(): for neighbor, weight in neighbors.items(): G.add_edge(node, neighbor, weight=weight) pos = nx.spring_layout(G) nx.draw(G, pos) labels = nx.get_edge_attributes(G, 'weight') nx.draw_networkx_edge_labels(G, pos, edge_labels=labels) nx.draw_networkx_labels(G, pos) plt.show()
Question: I have 2 dataframes of number x and y of same length and an input number a. I would like to find the fastest way to calculate a third list z such as :  z[0] = a z[i] = z[i-1]*(1+x[i]) + y[i]  without using a loop like that : a = 213 x = pd.DataFrame({'RandomNumber': np.random.rand(200)}) y = pd.DataFrame({'RandomNumber': np.random.rand(200)}) z = pd.Series(index=x.index, dtype=float) z[0] = a for i in range(1,len(x.index)): z[i] = z[i-1]*(1+x.iloc[i]) + y.iloc[i] 

Answer: You can't really vectorize this function, the development of the operation becomes too complex. For instance, z[i+1] expressed in function of z[i-1] is equal to: z[i-1]*(1+x[i])+y[i] + z[i]*(x[i+1]+x[i]*x[i+1]) + y[i]*x[i] + y[i+1]  And this get worse for each step As suggested in comment, if speed is a concern, you could use numba: from numba import jit @jit(nopython=True) def f(a, x, y): out = [a] for i in range(1, len(x)): out.append(out[i-1] * (1 + x[i]) + y[i]) return out out = pd.Series(f(213, x['RandomNumber'].to_numpy(), y['RandomNumber'].to_numpy()), index=x.index)  Output (using np.random.seed(0) and with 5 rows): 0 213.000000 1 365.772922 2 587.139217 3 908.025165 4 1293.097825 dtype: float64  Timings (200 rows): # python loop 59.8 ms ± 908 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) # numba 151 µs ± 968 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
Question: Currently I have my code set up like this: def lregression(data, X, y): X = df['sales'].values.reshape(-1, 1) y = df['target'] model = LinearRegression() result = model.fit(X, y) return model.score(X, y)  Then, I'm trying to apply this model per brand: df.groupby('brand').apply(lregression, X, y)  But the result just gets applied to the full dataset: Brand A 0.734 Brand B 0.734 Brand C 0.734  Am I missing something here? I want the model to run separately for each group, but instead I'm apparently getting the model applied to the full dataset and then having the overall score returned for each group. Thanks!

Answer: DATAFRAME A minimal reproducible example is always nice to have, I'll provide it here: np.random.seed(42) data = { 'brand': np.random.choice(['Brand A', 'Brand B', 'Brand C'], size=300), 'sales': np.random.randint(100, 1000, size=300), 'target': np.random.randint(100, 1000, size=300) } df = pd.DataFrame(data)  FUNCTION To me it's not clear whether you want to return the score (namely R^2) or the coef of the single regressions, in both cases the function changes only slightly:  Score  def lregression(group): X = group['sales'].values.reshape(-1, 1) y = group['target'] model = LinearRegression() result = model.fit(X, y) return result.score(X, y)   Coefficients  def lregression(group): X = group['sales'].values.reshape(-1, 1) y = group['target'] model = LinearRegression() result = model.fit(X, y) return result.coef_  Then the final step (coef_ scenario): >>> df.groupby('brand').apply(lregression) brand Brand A [0.20322970187699263] Brand B [0.09134770152569331] Brand C [0.043343302335992005] dtype: object  Which works as expected
Question: How do I use polars.Expr.list.sort to sort a list of structs by one of the struct values, i.e. df = pl.DataFrame([{"id": 1, "data": [{"key": "A", "value": 2}, {"key": "B", "value": 1}]}])  I want to sort data by the value field, i.e. the result should be df = pl.DataFrame([{"id": 1, "data": [{"key": "B", "value": 1}, {"key": "A", "value": 2}]}])  df.with_columns(pl.col("b").list.sort())  Doesn't work and list.sort doesn't accept arguments?

Answer: Indeed, pl.Expr.list.sort does not take any arguments on the comparison function used for the sorting. This seems like a reasonable feature request for polars' Github page. Until this is implemented, I can think of 2 possible approaches.  Polars seems to sort the list primarily based on the first struct field. Hence, reordering the struct fields before sorting influences the sorting behaviour. ( df .with_columns( pl.col("data") .list.eval( pl.struct( pl.element().struct.field("value"), pl.element().struct.field("key"), ) ) .list.sort() ) )  shape: (1, 2) ┌─────┬────────────────────┐ │ id ┆ data │ │ --- ┆ --- │ │ i64 ┆ list[struct[2]] │ ╞═════╪════════════════════╡ │ 1 ┆ [{1,"B"}, {2,"A"}] │ └─────┴────────────────────┘   Another more general approach is to explode the list column into multiple rows. Next, the frame can be sorted by the "value" field of the struct column "data" within each group defined by the column "id". This method offers a bit more flexibility as the sorting can be done based on multiple struct fields (to handle matches) and with mixed values for the descending parameter of pl.DataFrame.sort. ( df .explode("data") .sort("id", pl.col("data").struct.field("value")) .group_by("id") .agg("data") )  shape: (1, 2) ┌─────┬────────────────────┐ │ id ┆ data │ │ --- ┆ --- │ │ i64 ┆ list[struct[2]] │ ╞═════╪════════════════════╡ │ 1 ┆ [{"B",1}, {"A",2}] │ └─────┴────────────────────┘
Question: From the documentation for numpy's random.Generator.choice function, one of the arguments is shuffle, which defaults to True. The documentation states:  shuffle bool, optional Whether the sample is shuffled when sampling without replacement. Default is True, False provides a speedup.  There isn't enough information for me to figure out what this means. I don't understand why we would shuffle if it's already appropriately random, and I don't understand why I would be given the option to not shuffle if that yields a biased sample. If I set shuffle to False am I still getting a random (independent) sample? I'd love to also understand why I would ever want the default setting of True.

Answer: You are still getting a random choice regardless of your selection for shuffle. If you select shuffle=False, however, the ordering of the output is not independent of the ordering of the input. This is easiest to see when the number of items chosen equals the total number of items: import numpy as np rng = np.random.default_rng() x = np.arange(10) rng.choice(x, 10, replace=False, shuffle=False) # array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) rng.choice(x, 10, replace=False, shuffle=True) # array([8, 1, 3, 9, 6, 5, 0, 7, 4, 2])  If you reduce the number of items chosen and use shuffle=False, you can confirm that which item(s) are missing is distributed as expected. import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng() x = np.arange(10) set_x = set(x) missing = [] for i in range(10000): # By default, all `p` are equal, so which item is # missing should be uniformly distributed y = rng.choice(x, 9, replace=False, shuffle=False) set_y = set(y) missing.append(set_x.difference(set_y).pop()) plt.hist(missing)   But you'll see that items that appeared earlier in x tend to appear earlier in the output and vice-versa. That is, the input and output orders are correlated. x = np.arange(10) correlations = [] for i in range(10000): y = rng.choice(x, 9, replace=False, shuffle=False) correlations.append(stats.spearmanr(np.arange(9), y).statistic) plt.hist(correlations)   If that is ok for your application, feel free to set shuffle=False for a speedup. %timeit rng.choice(10000, 5000, replace=False, shuffle=True) # 187 µs ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) %timeit rng.choice(10000, 5000, replace=False, shuffle=False) # 146 µs ± 18.4 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  The more items that are to be chosen, the more pronounced the speedup. %timeit rng.choice(10000, 1, replace=False, shuffle=True) # 17.6 µs ± 3.64 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) %timeit rng.choice(10000, 1, replace=False, shuffle=False) # 16.5 µs ± 2.47 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)  vs %timeit rng.choice(10000, 9999, replace=False, shuffle=True) # 214 µs ± 32.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) %timeit rng.choice(10000, 9999, replace=False, shuffle=False) # 124 µs ± 27.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Question: >>> import sys >>> del sys.modules['sys'] >>> import sys >>> sys.modules Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'sys' has no attribute 'modules'  Why does re-imported sys module not have some attributes anymore? I am using Python 3.12.3 and it happens in macOS, Linux, and Windows. It happens in both the REPL and in a .py script. It does not happen in Python 3.11.

Answer: This is pretty obviously something you shouldn't do, naturally liable to break things. It happens to break things this particular way on the Python implementation you tried, but Python doesn't promise what will happen. Most of what I am about to say is implementation details.  The sys module cannot be initialized like normal built-in modules, as it's responsible for so much core functionality. Instead, on interpreter startup, Python creates the sys module with the special function _PySys_Create. This function is responsible for (part of the job of) correctly setting up the sys module, including the sys.modules attribute:  if (PyDict_SetItemString(sysdict, "modules", modules) < 0) { goto error; }  When you do del sys.modules['sys'], the import system loses track of the sys module. When you try to import it again, Python tries to create an entirely new sys module, and it does so as if sys were an ordinary built-in module. It goes through the procedure for initializing ordinary built-in modules. This procedure leaves the new sys module in an inconsistent, improperly initialized state, as sys was never designed to be initialized this way. There is support for reloading sys, although I believe the dev team is thinking of taking this support out - the use cases are very obscure, and the only one I can think of off the top of my head is obsolete. Part of the reinitialization ends up hitting a code path intended for reloading sys, which updates its __dict__ from a copy created early in the original initialization of sys, right before sys.modules is set:  interp->sysdict_copy = PyDict_Copy(sysdict); if (interp->sysdict_copy == NULL) { goto error; } if (PyDict_SetItemString(sysdict, "modules", modules) < 0) { goto error; }  This copy is handled differently on earlier Python versions, hence the version-related behavior differences.
Question: Supposing we have a class as follows: class PersonalChef(): def cook(): print("cooking something...")  And we want what it cooks to be a function of the time of day, we could do something like this: class PersonalChef(): def cook(time_of_day): ## a few ways to do this, but this is quite concise: meal = {'morning':'breakfast', 'midday':'lunch', 'evening':'dinner'}[time_of_day] print("Cooking", meal) PersonalChef().cook('morning') >>> Cooking breakfast  A potentially nice syntax form for this would be using decorators. With some under-the-hood machinery buried inside at_time, it ought to be possible to get it to work like this: class PersonalChef(): @at_time('morning') def cook(): print("Cooking breakfast") @at_time('midday') def cook(): print("Cooking lunch") @at_time('evening') def cook(): print("Cooking dinner") PersonalChef().cook('morning') >>> Cooking breakfast  The reason this could be a nice syntax form is shown by how it then shows up in subclasses: class PersonalChefUK(PersonalChef): @at_time('evening') def cook(): print("Cooking supper")  The code written at the sub-class level is extremely minimal and doesn't require any awareness of the base-class implementation/data structures and doesn't require any calls to super() to pass-through the other scenarios. So it could be nice in a situation where there are a large number of people writing derived-classes for whom we want to pack and hide complexity away in the base class and make it hard for them to break the functionality. However, I've tried a few different ways of implementing this and gotten stuck. I'm quite new to decorators, though, so probably missing something important. Any suggestions/comments?

Answer: A simpler approach would be to create a decorator factory that stores the decorated function in a dict that maps the function name and time of day to the function object, and returns a wrapper function that calls the stored function in the first class in the MRO that has it defined for the given time of day: def at_time(time_of_day, _actions={}): def decorator(func): def wrapper(self, time_of_day): for cls in type(self).__mro__: if func := _actions.get((f'{cls.__qualname__}.{name}', time_of_day)): return func(self) raise ValueError(f'No {name} found in the {time_of_day} time') name = func.__name__ _actions[func.__qualname__, time_of_day] = func return wrapper return decorator  so that: class PersonalChef(): @at_time('morning') def cook(self): print("Cooking breakfast") @at_time('evening') def cook(self): print("Cooking dinner") class PersonalChefUK(PersonalChef): @at_time('evening') def cook(self): print("Cooking supper") PersonalChef().cook('morning') PersonalChef().cook('evening') PersonalChefUK().cook('morning') PersonalChefUK().cook('evening')  outputs: Cooking breakfast Cooking dinner Cooking breakfast Cooking supper  Demo: https://ideone.com/9T94sq