eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have been reading this article A Simple Guide for Plotting a Proper Bifurcation Diagram and I want to reproduce the following figure (Fig. 10, p. 2150011-7):  I have created this procedure that works fine with other classical bifurcation diagrams: def model(x, r): return 8.821 * np.tanh(1.487 * x) - r * np.tanh(0.2223 * x) def diagram(r, x=0.1, n=1200, m=200): xs = [] for i in range(n): x = model(x, r) if i >= n - m: xs.append(x) return np.array(xs).T rlin = np.arange(5, 30, 0.01) xlin = np.linspace(-0.1, 0.1, 2) clin = np.linspace(0., 1., xlin.size) colors = plt.get_cmap("cool")(clin) fig, axe = plt.subplots(figsize=(8, 6)) for x0, color in zip(xlin, colors): x = diagram(rlin, x=x0, n=600, m=100) _ = axe.plot(rlin, x, ',', color=color) axe.set_title("Bifurcation diagram") axe.set_xlabel("Parameter, $r$") axe.set_ylabel("Serie term, $x_n(r)$") axe.grid()  But for this system, it renders:  Which looks similar to some extent but is not at the same scale and when r > 17.5 has a totally different behaviour than presented in the article. I am wondering why this difference happens. What have I missed?

Answer: I think that the journal reviewers for that article should have been a little more careful. The model equation is based on an earlier article (Baghdadi et al., 2015 - I had to go through my workplace's institutional access to get at it) the original value of B is 5.821, not 8.821 (see Fig 2 in the 2015 article). def model(x, r): return 5.821 * np.tanh(1.487 * x) - r * np.tanh(0.2223 * x)  This renders as
Question: Let's say I have a list of people ['foo','bar','baz']  and a list of items ['hat','bag','ball','bat','shoe','stick','pie','phone']  and I want to randomly assign each person an equal amount of items, like so { 'foo':['hat','bat','stick'], 'bar':['bag','shoe','phone'], 'baz':['ball','pie'] }  I think itertools is the job for this, but I couldn't seem to find the right function as most itertools functions seem to just work on one object. EDIT: Order does not matter. I just want to randomly assign each person an equal amount of items.

Answer: Another solution, using itertools.cycle: import random from itertools import cycle persons = ["foo", "bar", "baz"] items = ["hat", "bag", "ball", "bat", "shoe", "stick", "pie", "phone"] random.shuffle(items) out = {} for p, i in zip(cycle(persons), items): out.setdefault(p, []).append(i) print(out)  Prints (for example): { "foo": ["phone", "pie", "bat"], "bar": ["bag", "stick", "hat"], "baz": ["shoe", "ball"], }   If there could be fewer items than persons and each person should have key in output dictionary you can use: import random from itertools import cycle persons = ["foo", "bar", "baz"] items = ["hat", "bag", "ball", "bat", "shoe", "stick", "pie", "phone"] random.shuffle(items) random.shuffle(persons) # to randomize who gets fewest items out = {p: [] for p in persons} for lst, i in zip(cycle(out.values()), items): lst.append(i) print(out)
Question: What is the equivalent of pandas.DataFrame.query in polars? import pandas as pd data= { 'A':["Polars","Python","Pandas"], 'B' :[23000,24000,26000], 'C':['30days', '40days',np.nan], } df = pd.DataFrame(data) A B C 0 Polars 23000 30days 1 Python 24000 40days 2 Pandas 26000 NaN  Now, defining a variable item item=24000 df.query("B>= @item") A B C 1 Python 24000 40days 2 Pandas 26000 NaN  Now, using polars: import polars as pl df = pl.DataFrame(data) item=24000 df.query("B>= @item")  I get: AttributeError: 'DataFrame' object has no attribute 'query'  My wild guess is df.filter() but syntax does not look same and filter looks equivalent to df.loc[] as well ?

Answer: indeed, filter is all you need df.filter(pl.col("B") >= item)  clean, simple, predictable, without hacks
Question: Say, I have a dataset in the following DataFrame: df=pl.DataFrame({ 'x':['a','a','b','b'], 'y':['b','c','c','a'], 'value':[3,5,1,4] }) df shape: (4, 3) ┌─────┬─────┬───────┐ │ x ┆ y ┆ value │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 │ ╞═════╪═════╪═══════╡ │ a ┆ b ┆ 3 │ │ a ┆ c ┆ 5 │ │ b ┆ c ┆ 1 │ │ b ┆ a ┆ 4 │ └─────┴─────┴───────┘  Now, I'd like to add a column to this dataframe that would contain the inverse value. I define an inverse value as inverse(x, y) == value(y, x). E.g., from the example above, inverse (a, b) == value(b, a) == 4. If value(y, x) didn't exist then inverse(x, y) would be given the default value of 0. In other words, I'd like to add an inverse column such as I'd end up with something like this: shape: (4, 4) ┌─────┬─────┬───────┬─────────┐ │ x ┆ y ┆ value ┆ inverse │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 ┆ i64 │ ╞═════╪═════╪═══════╪═════════╡ │ a ┆ b ┆ 3 ┆ 4 │ │ a ┆ c ┆ 5 ┆ 0 │ │ b ┆ c ┆ 1 ┆ 0 │ │ b ┆ a ┆ 4 ┆ 3 │ └─────┴─────┴───────┴─────────┘  Is this doable in an easy and optimal way? Preferably with expressions? Thanks a lot in advance.

Answer: You could join it to itself with aliases and then do fill_null(0). df.join( df.select( y="x", x="y", inverse="value" ), on=["x","y"], how="left" ).fill_null(0) shape: (4, 4) ┌─────┬─────┬───────┬─────────┐ │ x ┆ y ┆ value ┆ inverse │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 ┆ i64 │ ╞═════╪═════╪═══════╪═════════╡ │ a ┆ b ┆ 3 ┆ 4 │ │ a ┆ c ┆ 5 ┆ 0 │ │ b ┆ c ┆ 1 ┆ 0 │ │ b ┆ a ┆ 4 ┆ 3 │ └─────┴─────┴───────┴─────────┘
Question: I know that Pydantic creates a deepcopy of mutable objects for "every new instances" that we create, if they are placed in the default values. It holds true for my lst field, but not for my custom object item. (The code for __deepcopy__ is taken from here) from copy import deepcopy from typing import Self from pydantic import BaseModel, ConfigDict class Spam: def __init__(self) -> None: self.names = ["hi"] def __deepcopy__(self, memo: dict) -> Self: print("deepcopy called") cls = self.__class__ result = cls.__new__(cls) memo[id(self)] = result for k, v in self.__dict__.items(): setattr(result, k, deepcopy(v, memo)) return result class Person(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) item: Spam = Spam() lst: list = [] print("-----------------------------------") obj1 = Person() obj2 = Person() obj1.lst.append(10) obj1.item.names.append("bye") print(obj1.lst) print(obj1.item.names) print(obj2.lst) print(obj2.item.names) print(id(obj1.item) == id(obj2.item))  output: deepcopy called ----------------------------------- [10] ['hi', 'bye'] [] ['hi', 'bye'] True  I printed the dashed-line after creating the class and before any instantiation just to show that the deepcopy of my object is indeed happened in class creation which is in contrast to the documentation:  Pydantic will deepcopy the default value when creating each instance of the model  Do I miss something here?

Answer: It works only for not hashable objects, but according to Python documentation  Objects which are instances of user-defined classes are hashable by default  https://docs.python.org/3/glossary.html#term-hashable (I was also surprised) So, you can use default_factory to achieve your goal: from copy import deepcopy from pydantic import BaseModel, ConfigDict, Field class Spam: def __init__(self) -> None: self.names = ["hi"] # def __deepcopy__(self, memo: dict): # print("deepcopy called") # cls = self.__class__ # result = cls.__new__(cls) # memo[id(self)] = result # for k, v in self.__dict__.items(): # setattr(result, k, deepcopy(v, memo)) # return result class Person(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) item: Spam = Field(default_factory=Spam) lst: list = [] print("-----------------------------------") obj1 = Person() obj2 = Person() obj1.lst.append(10) obj1.item.names.append("bye") print(obj1.lst) print(obj1.item.names) print(obj2.lst) print(obj2.item.names) print(id(obj1.item) == id(obj2.item))
Question: There seems to be some ambiguity in how the use of | in attribute type annotations together with attribute default values is parsed. Leading to the following error: >>> class A: pass ... >>> class B: ... A: A | None = None ... Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in B TypeError: unsupported operand type(s) for |: 'NoneType' and 'NoneType'  If we rename the class A to something else it works. Why does this not parse correctly?

Answer: A class definition creates a namespace, and by placing A on the left side of an assignment the compiler binds A as a local name to the None object in the namespace of the class B, instead of looking up A outside the scope of the class B and binding A to the class A defined in the module namespace. So A | None gets evaluated as None | None, and since the type of None, NoneType, doesn't support the | operation, you get the said TypeError.
Question: I am trying to bin values to prepare data to be later fed into a plotting library. For this I am trying to use polars Expr.cut. The dataframe I operate on contains different groups of values, each of these groups should be binned using different breaks. Ideally I would like to use np.linspace(BinMin, BinMax, 50) for the breaks argument of Expr.cut. I managed to make the BinMin and BinMax columns in the dataframe. But I can't manage to use np.linspace to define the breaks dynamically for each row of the dataframe. This is a minimal example of what I tried: import numpy as np import polars as pl df = pl.DataFrame({"Value": [12], "BinMin": [0], "BinMax": [100]})  At this point the dataframe looks like: ┌───────┬────────┬────────┐ │ Value ┆ BinMin ┆ BinMax │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═══════╪════════╪════════╡ │ 12 ┆ 0 ┆ 100 │ └───────┴────────┴────────┘  And trying to use Expr.cut with dynamic breaks: df.with_columns(pl.col("Value").cut(breaks=np.linspace(pl.col("BinMin"), pl.col("BinMax"), 50)).alias("Bin")) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[10], line 1 ----> 1 df.with_columns(pl.col("Value").cut(breaks=range(pl.col("BinMin"), pl.col("BinMax"))).alias("Bin")) TypeError: 'Expr' object cannot be interpreted as an integer  I understand the error, that np.linspace is expecting to be called with actual scalar integers, not polars Expr. But I cannot figure out how to call it with dynamic breaks derived from the BinMin and BinMax columns.

Answer: Unfortunately, pl.Expr.cut doesn't support expressions for the breaks argument (yet), but requires a fixed sequence. (This would be a good feature request though). A naive solution that will work for DataFrames, but doesn't use polars' native expression API, would be to use pl.Expr.map_elements together with the corresponding functionality in numpy. def my_cut(x, num=50): seq = np.linspace(x["BinMin"], x["BinMax"], num=num) idx = np.digitize(x["Value"], seq) return seq[idx-1:idx+1].tolist() ( df .with_columns( pl.struct("Value", "BinMin", "BinMax").map_elements(my_cut).alias("Bin") ) )  shape: (1, 4) ┌───────┬────────┬────────┬────────────────────────┐ │ Value ┆ BinMin ┆ BinMax ┆ Bin │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ list[f64] │ ╞═══════╪════════╪════════╪════════════════════════╡ │ 12 ┆ 0 ┆ 100 ┆ [10.204082, 12.244898] │ └───────┴────────┴────────┴────────────────────────┘
Question: I have 2 dataframes. For each row in dataframe2, I want to see if dataframe1 already has a row with the same value in column 'name'. If it does, I want to add the data from the row in dataframe2 to corresponding row in dataframe 1. The value in column 'name' should not be added across. Steve should not be copied across and bob should have no new data added.  df1 = pd.DataFrame([{'name': 'Ben', 'goals': 1, 'minutes': 90}, {'name': 'Bob', 'goals': 1, 'minutes': 64}, {'name': 'Kevin', 'goals': 1, 'minutes': 90}]) df2 = pd.DataFrame([{'name': 'Ben', 'goals': 1, 'minutes': 88}, {'name': 'Kevin', 'goals': 1, 'minutes': 3}, {'name': 'Steve', 'goals': 1, 'minutes': 13}])  The final output should be:    name goals minutes     Ben 2 178   Bob 1 64   Kevin 2 93    This is what I have tried for index, row in df1.iterrows(): if df2.isin([row['name']]).any().any(): position = int(df2[df2['name'] == str(row['name'])].index.values) df1.iloc[index, 1:] = df1.iloc[index, 1:] + df2.iloc[position, 1:] 

Answer:  Concatenate df1 + df2 minus the rows with names that aren't in df1; Group by name; Sum the values in each group; Reset the index.  df = pd.concat([df1, df2.loc[df2["name"].isin(df1["name"])]]).groupby("name").sum().reset_index()  Result:  name goals minutes 0 Ben 2 178 1 Bob 1 64 2 Kevin 2 93
Question: import polars as pl df = pl.DataFrame( { "X": [4, 2, 3, 4], "Y": ["p", "p", "p", "p"], "Z": ["b", "b", "b", "b"], } )  We know the equivalent of pandas's df.drop_duplicates() is df.unique() in python-polars But, each time I execute my query I get a different result? print(df.unique())   X Y Z i64 str str 3 "p" "b" 2 "p" "b" 4 "p" "b"   X Y Z i64 str str 4 "p" "b" 2 "p" "b" 3 "p" "b"   X Y Z i64 str str 2 "p" "b" 3 "p" "b" 4 "p" "b"  Is this intentional and what is the reason behind it?

Answer: Yes, this is an intentional behavior. If you need a consistent behavior then do: df.unique(maintain_order=True)  polars.DataFrame.unique maintain_order  Keep the same order as the original DataFrame. This is more expensive to compute. Settings this to True blocks the possibility to run on the streaming engine.  Maintaining order is not streaming-friendly as it requires bringing together all the chunks in memory to compare the order of the rows. With this change of default the developers want to ensure that Polars is ready to work with datasets of all sizes while allowing users to choose different behaviour if desired. A related point is the choice of which row within each duplicated group is kept by unique. In Pandas this defaults to the first row of each duplicated groups. In Polars the default is any as this again allows more optimizations. Other functions that have this behavior include: 1.group_by (maintain_order: bool = False) 2.partition_by (maintain_order: bool = True) 3.pivot (maintain_order: bool = True) 4.upsample (maintain_order: bool = False) A detailed article here by @LiamBrannigan: https://www.rhosignal.com/posts/polars-ordering/
Question: I have a many-column DF from which I need to process various ranges of columns. In Pandas I could use an expression along the lines of : df.loc[:, 'first_name': 'last_name'] to obtain the required columns between the two end-points. Is there an equivalent in Polars which does not involve listing all the numerous column names in each required range?

Answer: Since df.columns is just a list you can use the .index method to find their locations. Accomplishing this via the eager [] API or the .select interface can be done as follows: import polars as pl def inclusive(target, a, b): start, stop = target.columns.index(a), target.columns.index(b) return pl.col(target.columns[start:stop+1]) df = pl.DataFrame({ 'date': ['2000-01-01', '2000-01-02'], 'first_name': ['Alice', 'Bob' ], 'middle_name': [None, 'Edward' ], 'last_name': ['Smith', 'Jones' ], 'standing': ['good', 'bad'], }) print( df[:, 'first_name':'last_name'], # eager [] selection # shape: (2, 3) # ┌────────────┬─────────────┬───────────┐ # │ first_name ┆ middle_name ┆ last_name │ # │ --- ┆ --- ┆ --- │ # │ str ┆ str ┆ str │ # ╞════════════╪═════════════╪═══════════╡ # │ Alice ┆ null ┆ Smith │ # │ Bob ┆ Edward ┆ Jones │ # └────────────┴─────────────┴───────────┘ df.select(inclusive(df, 'first_name', 'last_name')), # shape: (2, 3) # ┌────────────┬─────────────┬───────────┐ # │ first_name ┆ middle_name ┆ last_name │ # │ --- ┆ --- ┆ --- │ # │ str ┆ str ┆ str │ # ╞════════════╪═════════════╪═══════════╡ # │ Alice ┆ null ┆ Smith │ # │ Bob ┆ Edward ┆ Jones │ # └────────────┴─────────────┴───────────┘ sep='\n\n', )
Question: I'm familiar with installing dependencies using a requirements.txt or environment.yml, but I've only ever seen syntax in those files like some_package>=1.2.3. What does it mean when dependencies are listed with curly braces, as in: pytest = "^6.2.5" coverage = {extras = ["toml"], version = "^5.5"} safety = "^1.10.3" mypy = "^0.910" typeguard = "^2.12.1" xdoctest = {extras = ["colors"], version = "^0.15.5"} Sphinx = "^4.1.2" sphinx-autobuild = "^2021.3.14"  and how do you install those dependences? Trying to install those by treating the file as a requirements.txt or environment.yml throws an ERROR: Invalid requirement or CondaValueError: invalid package specification, respectively.

Answer: python-poetry uses that format for defining dependencies in a pyproject.toml file usually under [tool.poetry.dependencies] As explained by PEP 508 packages can have extra dependencies which enables optional features of a given package (package dependent). You could install poetry and use that to manage dependencies or transform that list into a valid format that conda or pip understands. For example the dependency list for pip would look like this: pytest>=6.2.5 coverage[toml]>=5.5 safety>=1.10.3 mypy>=0.910 typeguard>=2.12.1 xdoctest[colors]>=0.15.5 Sphinx>=4.1.2 sphinx-autobuild>=2021.3.14
Question: I have a polars DataFrame: import polars as pl import numpy as np df = pl.DataFrame({'A': ['red', 'blue', 'green', np.nan, 'orange']}) my_list = ['red','orange']  I want to know which colors are present in my_list. In pandas, I would do something like: df.A.isin(my_list)  But, I am getting this error: AttributeError: 'DataFrame' object has no attribute 'A'  How to do this in polars?

Answer: The following can be used to create a dataset where column A is replaced by a boolean column indicating whether each item was in my_list. df.with_columns(pl.col("A").is_in(my_list))  shape: (5, 1) ┌───────┐ │ A │ │ --- │ │ bool │ ╞═══════╡ │ true │ │ false │ │ false │ │ null │ │ true │ └───────┘  If instead you'd like to  create a new boolean column, use pl.Expr.alias as follows df.with_columns(pl.col("A").is_in(my_list).alias("B")   have the return value be a pl.Series (like in @KrisKruse's answer), use pl.DataFrame.get_column as follows df.get_column("A").is_in(my_list)   filter for rows for which the value of A is in my_list, use pl.DataFrame.filter as follows df.filter(pl.col("A").is_in(my_list))
Question: I have the following function import math def f(k,a,x): return (1/(1+math.exp(-k*x))^a)  And the following pandas dataframe import pandas as pd df = pd.DataFrame({'x': list(range(1,100))})  How can I apply the above function on the x column of the df for lets say k=1 and a=2 ?

Answer: Using your function: def f(k,a,x): return (1/(1+math.exp(-k*x))**a) df['out'] = df['x'].apply(lambda x: f(k=1, a=2, x=x))  But better vectorize with numpy: import numpy as np def f_vectorized(k, a, x): return (1/(1+np.exp(-k*x))**a) df['out'] = f_vectorized(k=1, a=2, x=df['x'])  Output:  x out 0 1 0.534447 1 2 0.775803 2 3 0.907397 3 4 0.964351 4 5 0.986659 .. .. ... 94 95 1.000000 95 96 1.000000 96 97 1.000000 97 98 1.000000 98 99 1.000000 [99 rows x 2 columns]
Question: I would like to have the last character of a fact. My playbook looks like this: - name: sf set_fact: root_dev: "{{ ansible_mounts|json_query('[?mount == `/`].device') }}" - name: e debug: msg: "{{ root_dev[:-1] }}"  The problem is the output in this case always: "msg": []  or if I use the playbook without semicolon:  debug: msg: "{{ root_dev[-1] }}"  then the whole partition will be the output: "msg": "/dev/sda1"  I also can't quote the whole root_dev because it is a fact and I would like to get the last character of it's value. The split filter also not working in this case, because the device can be /dev/sda or /dev/mapper/root_part_0 and so. What would be the best option in this case?

Answer: Explanation of the problem: You have to understand how the slicing works. Given the lists l1 and l2 for testing  l1: [a] l2: [a, b, c]  The index [-1] gives you the last item (which is also the first item)  l1[-1]: a  The slice [:-1] gives you all items but the last one. The result is an empty list because there is only one item l1[:-1]: []  Test it by slicing l2  l2[:-1]: [a, b]   Solution: The value of root_dev is a list  root_dev: "{{ ansible_mounts | json_query('[?mount == `/`].device') }}"  In your case  root_dev: ['/dev/sda1']  To get the number of the partition, index the first item in the list and the last character in the item  root_dev[0][-1]: 1     Example of a complete playbook for testing  - hosts: localhost vars: root_dev: "{{ ansible_mounts | json_query('[?mount == `/`].device') }}" tasks: - setup: gather_subset: mounts # - debug: # var: ansible_mounts - debug: var: root_dev - debug: var: root_dev[0][-1]    The above solution works for a single-digit partitions only. Use regex_replace to cover also multiple-digit options   part_no: "{{ root_dev.0 | regex_replace('^(.+?)(\\d*)$', '\\2') }}"
Question: In pandas I would like to have a dataframe whose some columns have a multi index, some don't. Visually I would like something like this:  | c | | |--------| d | | a | b | | ================| | 1 | 4 | 0 | | 2 | 5 | 1 | | 3 | 6 | 2 |  In pandas I can do something like this: df = pd.DataFrame({'a':[1,2,3],'b':[4,5,6], 'd':[0,1,2]}) columns=[('c','a'),('c','b'), 'd'] df.columns=pd.MultiIndex.from_tuples(columns)  and the output would be:  c d a b NaN 0 1 4 0 1 2 5 1 2 3 6 2  However, when accessing the d column by df['d'], I get as output a Pandas Dataframe, not Pandas series. The problem is clearly that pandas applied multicolumn indexing to every column. So my question is: is there a way to apply column multindexing only to certain columns and leave the others as they are? In other words, I would like that the result of df['d'] would be a Series as in a normal dataframe, the result of df['c'] a pd.DataFrame as in column multindex and the result of df['c']['a'] a Pandas Series. Is this possible?

Answer: You can use the empty string "" as a placeholder : columns = [ ("c", "a"), ("c", "b"), ("d", ""), # << here ]  Output : type(df["d"]) # pandas.core.series.Series type(df["c"]) # pandas.core.frame.DataFrame type(df["c"]["a"]) # pandas.core.series.Series
Question: I want to change the value of a button written in html to "records". I want to adress and change the button in python. This is in order to prevent to tap the record button twice as the request form changes and to get a visual feedback that it works. this is what i have: @app.route('/camera', methods=['GET', 'POST']) def camera(): if request.method == 'POST': if request.form['record'] == 'record here' : # <-------- here I ask what the value of the button print ("is recording") # is and it works fine p1 = threading.Thread(target=recording) p1.start() return render_template('camera.html')  this is what i want: @app.route('/camera', methods=['GET', 'POST']) def camera(): if request.method == 'POST': if request.form['record'] == 'record here' : print ("is recording") p1 = threading.Thread(target=recording) p1.start() request.form['record'] = 'records' : # <----------------- Something like this return render_template('camera.html')  I am a newbie in python html flask and web development in general.

Answer: What you are looking to do should be done through JavaScript, because the most natural way to change the label of the button is through client-side programming. However, if you insist on doing this from server-side code, you can start by changing the template camera.html. Let's say that you have the following button in your template: <input type="submit" name="record" value="record here">  This should be changed to include a variable: <input type="submit" name="record" value="{{ record }}">  Now you can send the value of this variable using render_template: @app.route('/camera', methods=['GET', 'POST']) def camera(): if request.method == 'POST': if request.form['record'] == 'record here' : print ("is recording") p1 = threading.Thread(target=recording) p1.start() return render_template('camera.html', record='records') return render_template('camera.html', record='record here')