eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have time series data that looks like this (mm/dd hh:mm): 3.100 12/14 05:42 3.250 12/14 05:24 3.300 12/14 05:23 3.600 12/14 02:45 3.700 12/13 10:54 3.600 12/12 13:19 3.900 12/12 10:43  I need to interpolate it at 1 minute intervals. It will be a step chart, so the values should be the same until the new value.

Answer: If your goal is to make a step plot, no need to interpolate, just use matplotlib.pyplot.step: import matplotlib.pyplot as plt s = pd.Series(['12/14 05:42', '12/14 05:24', '12/14 05:23', '12/14 02:45', '12/13 10:54', '12/12 13:19', '12/12 10:43'], index=[3.1, 3.25, 3.3, 3.6, 3.7, 3.6, 3.9]) plt.step(pd.to_datetime(s, format='%m/%d %H:%M'), s.index)  NB. assuming here the values are the index and the dates the series' values, which is a bit counterintuitive. Better use the date as index. Output:  Also, be aware that without a year, the default will be to use 1900 during the conversion to datetime, which might be unwanted. Better be explicit and add the exact year.  If you really want to interpolate, use the date as index and asfreq: s = pd.Series([3.1, 3.25, 3.3, 3.6, 3.7, 3.6, 3.9], index=['12/14 05:42', '12/14 05:24', '12/14 05:23', '12/14 02:45', '12/13 10:54', '12/12 13:19', '12/12 10:43']) s.index = pd.to_datetime(s.index, format='%m/%d %H:%M') out = s.asfreq('min', method='ffill')  Output: 1900-12-12 10:43:00 3.9 1900-12-12 10:44:00 3.6 1900-12-12 10:45:00 3.6 1900-12-12 10:46:00 3.6 1900-12-12 10:47:00 3.6 ... 1900-12-14 05:38:00 3.1 1900-12-14 05:39:00 3.1 1900-12-14 05:40:00 3.1 1900-12-14 05:41:00 3.1 1900-12-14 05:42:00 3.1 Freq: T, Length: 2580, dtype: float64
Question: I find this code very interesting. I modified the code a little to improve the question. Essentially, the code uses a DataFrame to format the style of another DataFrame using pd.style. t1 = pd.DataFrame({'x':[300,200,700], 'y':[100,300,200]}) t2 = pd.DataFrame({'x':['A','B','C'], 'y':['C','B','D']}) def highlight_cell(val, props=''): return props if val > 200 else '' t2.style.apply(lambda x: t1.map(highlight_cell, props='background-color:yellow'), axis=None)   But can anyone explain how the last line works? I couldn't find Pandas documentation that clarifies the behavior of df.map() inside another df.apply(). To me, the code reads like for each item in t1, apply highlight_cell() to the entire t2 at once, and then return the whole thing, as illustrated in this pseudocode. for x in all_items_in_t1: yield [highlight_cell(y) for y in all_items_in_t2]  However, the output is saying for each item in t1, apply highlight_cell() only to the corresponding item in t2 that has the same (x, y) location as that item in t1, like this. for x, y in zip(all_items_in_t1, all_items_in_t2): yield highlight_cell(y)  I'm still having trouble understanding this pattern because it seems a bit confusing. Can anyone explain it more clearly?

Answer: DataFrame.style.apply is used here, not DataFrame.apply. By using the parameter axis=None, the callable is applied once (not per cell) on the whole DataFrame. Since the callable is a lambda, this essentially means we run: t1.map(highlight_cell, props='background-color:yellow')  and use the output as format.  x y 0 background-color:yellow 1 background-color:yellow 2 background-color:yellow  Note that using DataFrame.map here is not needed (and inefficient), better go for a vectorial approach: t2.style.apply(lambda x: np.where(t1>200, 'background-color:yellow', ''), axis=None)
Question: I was trying this: tuple(map(tuple, tuple(((x,y) for x in range(5)) for y in range(3))))  I got this: (((0, 2), (1, 2), (2, 2), (3, 2), (4, 2)), ((0, 2), (1, 2), (2, 2), (3, 2), (4, 2)), ((0, 2), (1, 2), (2, 2), (3, 2), (4, 2)))  but I expect: (((0, 0), (1, 0), (2, 0), (3, 0), (4, 0)), ((0, 1), (1, 1), (2, 1), (3, 1), (4, 1)), ((0, 2), (1, 2), (2, 2), (3, 2), (4, 2))) 

Answer: You're forcing the evaluations in the wrong order. You're building a generator of generators, building a tuple out of the outer generator to build a tuple of generators, and then building tuples out of the inner generators. By the time you start working with the inner generators, the outer generator has finished iteration, so y is already 2, the value from the last iteration. This is the y value used the entire time you're iterating over the inner generators, so it's the y value used every time you evaluate (x, y). You need to call tuple on the inner generators as they're produced, to iterate over them and evaluate (x, y) before y changes: tuple(tuple((x, y) for x in range(5)) for y in range(3))
Question: The python documentation writes about _* "Not imported by from module import *." What do they mean with that? https://docs.python.org/3/reference/lexical_analysis.html#:~:text=_*,import%20*.

Answer: The documentation uses * as a wildcard, meaning it substitutes for anything, similar to the way wildcards work in the shell. So when it says _*, it means any identifier beginning with _. So when you do from module import *, it imports all the top-level names in the module except those that begin with _. When writing a module, you use _XXX names to create private variables/functions/classes.
Question: I use scipy version 1.14.1 to traverse the minimum spanning tree in depth-first order, but I do not understand some results, namely the predecessors returned by scipy are not correct. Here is an illustration for the following graph:  The following code import numpy as np from scipy.sparse import coo_matrix from scipy.sparse.csgraph import minimum_spanning_tree from scipy.sparse.csgraph import depth_first_order rows = np.array([0, 1, 2, 2, 4, 9, 2, 2, 10, 10, 8 ]) cols = np.array([1, 2, 3, 4, 9, 5, 6, 10, 11, 8, 7 ]) # construct undirected graph X = coo_matrix( (12,12)) X.col = np.concatenate( (rows, cols), axis=0) X.row = np.concatenate( (cols, rows), axis=0) X.data = np.ones(len(X.row)) # the minimum spanning tree is the graph itself tree = minimum_spanning_tree(X) print(tree) # traversing the graph print(depth_first_order(tree, i_start=0, directed=False, return_predecessors=True))  gives the minimum spanning tree (the graph itself in fact):  Coords Values (0, 1) 1.0 (1, 2) 1.0 (2, 3) 1.0 (2, 4) 1.0 (2, 6) 1.0 (2, 10) 1.0 (4, 9) 1.0 (5, 9) 1.0 (7, 8) 1.0 (8, 10) 1.0 (10, 11) 1.0  and the depth-first order: [ 0, 1, 2, 3, 4, 9, 5, 6, 10, 11, 8, 7] predecessors: [-9999, 0, 1, 2, 2, 9, 2, 8,10, 4, 2, 10] So it says that 9 has 9 as ancestor, but it is 4, and from that position on results are not coherent. Thanks for any help.

Answer: From the documentation, the function depth_first_order() returns the following two lists:  node_array ndarray, one dimension The depth-first list of nodes, starting with specified node. The length of node_array is the number of nodes reachable from the specified node. predecessors ndarray, one dimension Returned only if return_predecessors is True. The length-N list of predecessors of each node in a depth-first tree. If node i is in the tree, then its parent is given by predecessors[i]. If node i is not in the tree (and for the parent node) then predecessors[i] = -9999.  It basically implies that the second list predecessors returned has nothing to do with the first list node_array returned. Rather, the 2nd list is to be read as:  [π[i] if i ∈ G else -9999 for all i], where π[i] is parent of node i and the graph G is a (spanning) tree here.  The list predecessors = [-9999, 0, 1, 2, 2, 9, 2, 8,10, 4, 2, 10] is to be read as following  π[0] = -9999 (since node 0 has no parent in G) π[1] = 0 (parent of node 1 is node 0) π[2] = 1 π[3] = 2 π[4] = 2 π[[5] = 9 π[6] = 2 π[7] = 8 ...  The same result you can obtain with networkx.dfs_predecessors(), more explicitly as a dictionary, where key is a node and value is the parent of the node (obtained using dfs traversal): import networkx as nx nx.dfs_predecessors(tree, source=0) # {1: 0, 2: 1, 3: 2, 4: 2, 9: 4, 5: 9, 6: 2, 10: 2, 11: 10, 8: 10, 7: 8}
Question: Here is a script to simulate the issue I am facing: import pandas as pd import numpy as np data = { 'a':[1,2,1,1,2,1,1], 'b':[10,40,20,10,40,10,20], 'c':[0.3, 0.2, 0.6, 0.4, 0.5, 0.2, 0.8], 'd':[3, 1, 5, 1, 7, 2., 2.], } df = pd.DataFrame.from_dict(data) # I apply some custom function to populate column 'e'. # For demonstration, I am using a very simple function here. df['e']=df.apply(lambda x: x['c']<=0.3, axis=1) # This is the column I need to obtain using groupby and pipe/transform df['f']=[2., 1., 0., 2., 1., 2., 0.] print(df)  Output:  a b c d e f 0 1 10 0.3 3.0 True 2.0 1 2 40 0.2 1.0 True 1.0 2 1 20 0.6 5.0 False 0.0 3 1 10 0.4 1.0 False 2.0 4 2 40 0.5 7.0 False 1.0 5 1 10 0.2 2.0 True 2.0 6 1 20 0.8 2.0 False 0.0  The logic to be used to find column f is as follows: For each group of df.groupby(['a', 'b']):  select entries with True value for e. if there are any item in the selected array:  find entry with minimum d and return d (in real application, d needs to be manipulated in conjunction with other columns, and the result would be returned)   else:  return 0    What I have tried: def func(x): print(type(x)) print(x) print('-'*50) ind=np.where(x['e']) #<--- How can I implement this? if len(ind)>0: ind_min=np.argmin(x.iloc[ind]['d']) return x.iloc[ind[ind_min]]['d'] else: return 0 df['g']=df.groupby(['a', 'b']).pipe(func)  Output: <class 'pandas.core.groupby.generic.DataFrameGroupBy'> <pandas.core.groupby.generic.DataFrameGroupBy object at 0x000001B348735550> -------------------------------------------------- ... ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (3, 2) + inhomogeneous part.  I get the mentioned error on line: ind=np.where(x['e']) #<--- How can I implement this? So, how can apply np.where on a pandas.core.groupby.generic.DataFrameGroupBy object?

Answer: You're presenting the XY problem. Here's one approach: cond = df['c'] <= 0.3 df['f'] = ( df.assign(filtered_d=df['d'].where(cond)) .groupby(['a', 'b'])['filtered_d'] .transform('min') .fillna(0) )  Output:  a b c d e f 0 1 10 0.3 3.0 True 2.0 1 2 40 0.2 1.0 True 1.0 2 1 20 0.6 5.0 False 0.0 3 1 10 0.4 1.0 False 2.0 4 2 40 0.5 7.0 False 1.0 5 1 10 0.2 2.0 True 2.0 6 1 20 0.8 2.0 False 0.0  Explanation / Intermediate  First, apply Series.where to column 'd' based on the boolean series to keep only the values that we want to consider for min:  # df['d'].where(cond) 0 3.0 1 1.0 2 NaN 3 NaN 4 NaN 5 2.0 6 NaN Name: d, dtype: float64   Use df.assign to include the result (here as filtered_d) and apply df.groupby + groupby.transform to get 'min'. Finally, add Series.fillna and assign as a new column.   An alternative way to do this could be: cond = df['c'] <= 0.3 df['f'] = ( df.merge( df[cond] .groupby(['a', 'b'], as_index=False) .agg(f=('d', 'min')), on=['a', 'b'], how='left' ).assign(f=lambda x: x['f'].fillna(0)) )  Explanation  Use boolean indexing + df.groupby + groupby.agg + named aggregation. Merge with df via df.merge + chain df.assign to apply Series.fillna.
Question: This code not fill null values in column. I want to some fields to forward and backward fill nulls. import polars as pl df1 = pl.LazyFrame({ "dt": [ "2024-08-30", "2024-08-02", "2024-09-03", "2024-09-04" ], "df1": { "a": [0.1, 0.2, 0.3, 0.1], "b": [0, -1, 2, 1] }, }).with_columns( pl.col("dt").str.to_datetime("%Y-%m-%d") ) df2 = pl.LazyFrame({ "dt": [ "2024-08-29", "2024-08-30", "2024-09-02", "2024-09-03" ], "df2":{ "a": [100, 120, -80, 20], "b": [1, -2, 0, 0] }, }).with_columns( pl.col("dt").str.to_datetime("%Y-%m-%d") ) df = pl.concat([df1, df2], how="align") df = df.with_columns( *[ pl.col(c).struct.with_fields( pl.field("a").forward_fill().backward_fill(), pl.field("b").forward_fill().backward_fill(), ) for c in ["df1", "df2"] ] ) print(df.collect())  Null values ​​appear in the output. I would expect them to be forward and backward filled, but they aren't. ┌─────────────────────┬───────────┬───────────┐ │ dt ┆ df1 ┆ df2 │ │ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ struct[2] ┆ struct[2] │ ╞═════════════════════╪═══════════╪═══════════╡ │ 2024-08-02 00:00:00 ┆ {0.2,-1} ┆ null │ │ 2024-08-29 00:00:00 ┆ null ┆ {100,1} │ │ 2024-08-30 00:00:00 ┆ {0.1,0} ┆ {120,-2} │ │ 2024-09-02 00:00:00 ┆ null ┆ {-80,0} │ │ 2024-09-03 00:00:00 ┆ {0.3,2} ┆ {20,0} │ │ 2024-09-04 00:00:00 ┆ {0.1,1} ┆ null │ └─────────────────────┴───────────┴───────────┘  I would expected this output: ┌─────────────────────┬───────────┬───────────┐ │ dt ┆ df1 ┆ df2 │ │ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ struct[2] ┆ struct[2] │ ╞═════════════════════╪═══════════╪═══════════╡ │ 2024-08-02 00:00:00 ┆ {0.2,-1} ┆ {100,1} │ │ 2024-08-29 00:00:00 ┆ {0.2,-1} ┆ {100,1} │ │ 2024-08-30 00:00:00 ┆ {0.1,0} ┆ {120,-2} │ │ 2024-09-02 00:00:00 ┆ {0.1,0} ┆ {-80,0} │ │ 2024-09-03 00:00:00 ┆ {0.3,2} ┆ {20,0} │ │ 2024-09-04 00:00:00 ┆ {0.1,1} ┆ {20,0} │ └─────────────────────┴───────────┴───────────┘  How to do that and why null values in column aren't filled? Probably it could be by pl.col(c).forward_fill().backward_fill(), but what if I want only one field to be filled?

Answer: The reason .struct.with_fields doesn't do what you want is because structs still have outer nullability, and with_fields does not have a special case where the outer nullability is ignored if all fields are replaced. So instead of using with_fields to update fields, completely replace the struct column with a new one which resets the outer nullability: out = df.with_columns( *[ pl.struct( pl.col(c).struct.field("a").forward_fill().backward_fill(), pl.col(c).struct.field("b").forward_fill().backward_fill(), ).alias(c) for c in ["df1", "df2"] ] )
Question: I am encountering an error when running the poetry install command in my Python project. The error message is as follows: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: ... (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))  I have tried the following troubleshooting steps, but none have resolved the issue:  Verified my internet connection. Checked that I have the latest version of Poetry installed. Ensured my Git and OpenSSL installations are up-to-date. Confirmed that the repository URL is correct and accessible from a browser.  Despite all this, the error persists. It seems related to SSL or Git configuration, but I am unsure how to resolve it. Environment Details: OS: 14.7.1 (23H222) Python version: 3.11.9] Poetry version: 1.8.5 Git version: 2.45.2  Has anyone encountered this issue before or can provide guidance on how to resolve it?

Answer: Here’s how I resolved the issue: 1. Install or Update certifi in the Virtual Environment Make sure the certifi package is installed and up-to-date: pip install --upgrade certifi  Locate the cacert.pem File Use the following command to find where the cacert.pem file is located: python -m certifi  2. Copy the cacert.pem File to the Expected Location From the error message (I tried to clone the repo from github using https), I saw that the virtual environment was looking for the cacert.pem file here: /Users/user/.../certifi/cacert.pem If the file is missing, copy it from another working virtual environment. For example: mkdir -p /Users/user/.../certifi/ cp /path/to/working/venv/lib/python3.11/site-packages/certifi/cacert.pem /Users/user/.../cacert.pem  3. Retry the Command After copying the cacert.pem file to the correct location, I retried the poetry install command and it worked!.
Question: I have a view being inherited by APIView and I have only implemented the GET method in it. class MyView(APIView): def get(self, request, id): # do something  But when I call getattr(ClassInstance, "http_method_names", []) I get list of all the HTTP methods even though they are not implemented.

Answer: The .http_method_names [Django-doc] is an attribute that, by default lists all HTTP methods, so:  ["get", "post", "put", "patch", "delete", "head", "options", "trace"]   Normally it will always return that list, unless you override it. This can be useful if you inherit from a view that for example has defined a .post(…) method, but you don't want to expose that, like: class MyView(SomethingWithPostMethodView): http_method_names = ['get', 'head', 'options'] def get(self, request, id): # … pass Or you could, technically speaking, expand the http_method_names list with an custom method, although I would strongly advise not to do this. In order to find out what methods the view implements, you can use the _allowed_methods(…), which thus checks if the corresponding .get(…) method is allowed or not. You thus can not introspect the methods allowed with: getattr(ClassInstance, "http_method_names", [])  You can just create an instance and work with _allowed_methods(): ClassInstance()._allowed_method()  If ClassInstance is thus a reference to the type itself, or use ClassInstance._allowed_method() if it is a view object already.
Question: How can I get the source index of a text node in an HTML string? Tags have sourceline and sourcepos which is useful for this, but NavigableString does not have any directly-helpful properties like that (as far as I can find) I've thought about using def get_index(text_node: NavigableString) -> int: return text_node.next_element.sourcepos - len(text_node)  But this will not work perfectly because the length of the closing tag is unpredictable, e.g. >>> get_index(BeautifulSoup('<p>hello</p><br>', 'html.parser').find(text=True)) 7  Is incorrect, and '<p>hello</p >' is also valid HTML and will produce an even more incorrect result, and I'm not sure how to solve this kind of case using the tools I've found so far in BeautifulSoup. I would also be interested in an lxml or Python html module answer if they have simple solutions. Desired results: >>> get_index(BeautifulSoup('hello', 'html.parser').find(text=True)) 0 >>> get_index(BeautifulSoup('<p>hello</p><br>', 'html.parser').find(text=True)) 3 >>> get_index(BeautifulSoup('<!-- hi -->hello', 'html.parser').find(text=True)) 11 >>> get_index(BeautifulSoup('<p></p ><p >hello<br>there</p>', 'html.parser').find(text=True)) 12 >>> get_index(BeautifulSoup('<p></p ><p >hello<br>there</p>', 'html.parser').find_all(string=True)[1]) 21 

Answer: Using html.parser: class MyHTMLParser(HTMLParser): def handle_data(self, data: str): line, col = self.getpos() previous_lines = ''.join(html_string.splitlines(True)[:line - 1]) index = len(previous_lines) + col print(data, 'at', index) parser = MyHTMLParser() parser.feed(html_string)
Question: Assume, there is one DataFrame such as following import pandas as pd import numpy as np df = pd.DataFrame({'id':range(1,4), 'items':[['A', 'B'], ['A', 'B', 'C'], ['A', 'C']]}) df id items 1 [A, B] 2 [A, B, C] 3 [A, C]  Is there an efficient way to convert above DataFrame into the following (one-hot encoded columns)? Many Thanks in advance!  id items A B C 1 [A, B] 1 1 0 2 [A, B, C] 1 1 1 3 [A, C] 1 0 1 

Answer: SOLUTION 1 A possible solution, whose steps are:  First, the explode function is used to transform each item of a list-like to a row, replicating the index values.  Then, the to_numpy method converts the resulting dataframe to a numpy array, and .T transposes this array.  The crosstab function computes a simple cross-tabulation of factors, which, in this case, are the transposed columns of the exploded dataframe.  The reset_index method is used to reset the index of the dataframe, turning the index into a column named id.  Finally, the original dataframe df is merged with this transformed dataframe using the merge function.   df.merge( pd.crosstab(*df.explode('items').to_numpy().T) .reset_index(names='id'))   SOLUTION 2 Another possible solution, whose steps are:  First, the explode function is used to transform each item of a list-like to a row, replicating the index values.  Then, the pivot_table function is applied to reshape the data based on the unique values in the items column, aggregating the count of each id for every item. The fill_value=0 ensures that any missing combinations are filled with zeros.  The rename_axis method is used to remove the axis name for the columns.  Finally, reset_index is called to reset the index of the dataframe, turning the index into a column.  The original dataframe df is then merged with this transformed dataframe using the merge function.   df.merge( df.explode('items') .pivot_table(index='id', columns='items', values='id', aggfunc=len, fill_value=0) .rename_axis(None, axis=1).reset_index())   Output:  id items A B C 0 1 [A, B] 1 1 0 1 2 [A, B, C] 1 1 1 2 3 [A, C] 1 0 1
Question: I'm using a python code to plot 3D surface. However, the z-axis label get cut by the figure. Here is the code : import matplotlib.pyplot as plt import numpy as np fig = plt.figure(figsize=(12, 10), facecolor='lightblue') x = np.linspace(0, 10) y = np.linspace(0, 10) X, Y = np.meshgrid(x, y) for idx in range(4): Z = np.cos(X) - np.sin(np.pi*idx/4 * Y) ax3D = fig.add_subplot(2, 2, idx+1, projection='3d') ax3D.plot_surface(X, Y, Z, cmap="viridis") ax3D.set_zlabel("Title") plt.show()  The result : 3D surface plots Is it a possible to include the axis label in the axe ? Or to increase the space subplots and the figure ? I have tried different options such as : plt.subplots_adjust(left=0, bottom=0, right=0.8, top=0.7, wspace=0.5, hspace=0.2) or fig.tight_layout(); but nothing seems to resolve my problem.

Answer: One solution is to zoom out to decrease the size of each subplot (set_box_aspect). One can also play with the three angles that defines the view: elevation, azimuth, and roll (view_init). fig = plt.figure(figsize=(12/2, 10/2), facecolor='lightblue') x = np.linspace(0, 10) y = np.linspace(0, 10) X, Y = np.meshgrid(x, y) for idx in range(4): Z = np.cos(X) - np.sin(np.pi*idx/4 * Y) ax3D = fig.add_subplot(2, 2, idx+1, projection='3d') ax3D.view_init(elev=30, azim=70, roll=0) ax3D.set_box_aspect(aspect=(1,1,1), zoom=0.8) ax3D.plot_surface(X, Y, Z, cmap="viridis") ax3D.set_zlabel("Title") fig.tight_layout() plt.show()