eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here's some sample data import polars as pl df = pl.DataFrame( { "date": [ "2024-08-01", "2024-08-02", "2024-08-03", "2024-08-04", "2024-08-04", "2024-08-05", "2024-08-06", "2024-08-08", "2024-08-09", ], "type": ["A", "A", "A", "A", "B", "B", "B", "A", "A"], } ).with_columns(pl.col("date").str.to_date())  And my desired output would look something like this shape: (9, 3) ┌────────────┬──────┬───────────────┐ │ date ┆ type ┆ days_in_a_row │ │ --- ┆ --- ┆ --- │ │ date ┆ str ┆ i64 │ ╞════════════╪══════╪═══════════════╡ │ 2024-08-01 ┆ A ┆ 1 │ │ 2024-08-02 ┆ A ┆ 2 │ │ 2024-08-03 ┆ A ┆ 3 │ │ 2024-08-04 ┆ A ┆ 4 │ │ 2024-08-04 ┆ B ┆ 1 │ │ 2024-08-05 ┆ B ┆ 2 │ │ 2024-08-06 ┆ B ┆ 3 │ │ 2024-08-08 ┆ A ┆ 1 │ │ 2024-08-09 ┆ A ┆ 2 │ └────────────┴──────┴───────────────┘  Where my days_in_a_row counter gets reset upon a date gap greater than 1 day. What I've tried so far df.with_columns(days_in_a_row=pl.cum_count("date").over("type"))  Which gives me shape: (9, 3) ┌────────────┬──────┬───────────────┐ │ date ┆ type ┆ days_in_a_row │ │ --- ┆ --- ┆ --- │ │ date ┆ str ┆ u32 │ ╞════════════╪══════╪═══════════════╡ │ 2024-08-01 ┆ A ┆ 1 │ │ 2024-08-02 ┆ A ┆ 2 │ │ 2024-08-03 ┆ A ┆ 3 │ │ 2024-08-04 ┆ A ┆ 4 │ │ 2024-08-04 ┆ B ┆ 1 │ │ 2024-08-05 ┆ B ┆ 2 │ │ 2024-08-06 ┆ B ┆ 3 │ │ 2024-08-08 ┆ A ┆ 5 │ │ 2024-08-09 ┆ A ┆ 6 │ └────────────┴──────┴───────────────┘  Which is not resetting after the gap. I can't quite nail this one down. I've also tried variations with df .with_columns(date_gap=pl.col("date").diff().over("type")) .with_columns(days_in_a_row=(pl.cum_count("date").over("date_gap", "type")))  Which get's closer, but it still ends up not resetting where I'd want it shape: (9, 4) ┌────────────┬──────┬──────────────┬───────────────┐ │ date ┆ type ┆ date_gap ┆ days_in_a_row │ │ --- ┆ --- ┆ --- ┆ --- │ │ date ┆ str ┆ duration[ms] ┆ u32 │ ╞════════════╪══════╪══════════════╪═══════════════╡ │ 2024-08-01 ┆ A ┆ null ┆ 1 │ │ 2024-08-02 ┆ A ┆ 1d ┆ 1 │ │ 2024-08-03 ┆ A ┆ 1d ┆ 2 │ │ 2024-08-04 ┆ A ┆ 1d ┆ 3 │ │ 2024-08-04 ┆ B ┆ null ┆ 1 │ │ 2024-08-05 ┆ B ┆ 1d ┆ 1 │ │ 2024-08-06 ┆ B ┆ 1d ┆ 2 │ │ 2024-08-08 ┆ A ┆ 4d ┆ 1 │ │ 2024-08-09 ┆ A ┆ 1d ┆ 4 │ └────────────┴──────┴──────────────┴───────────────┘ 

Answer: What about first creating a secondary grouper based on the duration ≥ 1day? from datetime import timedelta (df.with_columns(group=pl.col("date").diff().gt(timedelta(days=1)) .fill_null(True).cum_sum().over("type")) .with_columns(days_in_a_row=pl.cum_count("date").over(["type", "group"])) )  Output: ┌────────────┬──────┬───────┬───────────────┐ │ date ┆ type ┆ group ┆ days_in_a_row │ │ --- ┆ --- ┆ --- ┆ --- │ │ date ┆ str ┆ u32 ┆ u32 │ ╞════════════╪══════╪═══════╪═══════════════╡ │ 2024-08-01 ┆ A ┆ 1 ┆ 1 │ │ 2024-08-02 ┆ A ┆ 1 ┆ 2 │ │ 2024-08-03 ┆ A ┆ 1 ┆ 3 │ │ 2024-08-04 ┆ A ┆ 1 ┆ 4 │ │ 2024-08-04 ┆ B ┆ 1 ┆ 1 │ │ 2024-08-05 ┆ B ┆ 1 ┆ 2 │ │ 2024-08-06 ┆ B ┆ 1 ┆ 3 │ │ 2024-08-08 ┆ A ┆ 2 ┆ 1 │ │ 2024-08-09 ┆ A ┆ 2 ┆ 2 │ └────────────┴──────┴───────┴───────────────┘
Question: In the docs for randrange(), it states:  Keyword arguments should not be used because they can be interpreted in unexpected ways. For example randrange(start=100) is interpreted as randrange(0, 100, 1).  If the signature is random.randrange(start, stop[, step]), why doesn't randrange(start=100) raise an error, since stop is not passed a value? Why would randrange(start=100) be interpreted as randrange(0, 100, 1)? I'm not trying to understand the design choice of whoever wrote the code, so much as understanding how it's even possible. I thought parameters without default values need to be passed arguments or else a TypeError would be raised.

Answer: This is because the code for randrange must also support the alternative signature: random.randrange(stop)  The implementation performs a check to see whether the call uses this signature or the other one: random.randrange(start, stop[, step])  It does so with this code:  istart = _index(start) if stop is None: if istart > 0: return self._randbelow(istart)  This makes sense when you call randrange(100), where it translates into a call of _randbelow(100), which is intended. But when you call randrange(start=100), then we get into the same flow, and the value of start is also interpreted as stop.
Question: I'm using argparse.ArgumentParser in python and giving users the option to add their own functions to my script. For that, I'm giving them a function that generates data that gets used in argparse's add_argument() function. I noticed that add_argument's signature is add_argument(self, *args, **kwargs) but when looking at the documentation in vscode I can see all the different arguments it can get (action, nargs, const, default, etc.).  How does python do it? As far as I know when you give the user **kwargs it is shown as **kwargs and that's it. I looked all over this site and the internet in general but I can't find the syntax used for it.

Answer: VSCode's language server is pulling from the stub file in the typeshed, where the current definition is: def add_argument( self, *name_or_flags: str, action: _ActionStr | type[Action] = ..., nargs: int | _NArgsStr | _SUPPRESS_T | None = None, const: Any = ..., default: Any = ..., type: _ActionType = ..., choices: Iterable[_T] | None = ..., required: bool = ..., help: str | None = ..., metavar: str | tuple[str, ...] | None = ..., dest: str | None = ..., version: str = ..., **kwargs: Any, ) -> Action: ...  Sidenote: It looks like you're using Pylance, which is adding some detail to the types. For example with _T@add_argument it's saying that _T is scoped to add_argument. I'm using Jedi which shows different details. Reference Editing Python in Visual Studio Code § Autocomplete and IntelliSense  Pylance is the default language server for Python in VS Code, and is installed alongside the Python extension to provide IntelliSense features. Pylance is based on Microsoft's Pyright static type checking tool, leveraging type stubs (.pyi files) and lazy type inferencing to provide a highly-performant development experience.  See also  Stub files - mypy documentation
Question: So a little stylistic thing I want to do is move some buttons to the east side of the window while having them all on the same row. Sounds easy enough and the grid method has a sticky parameter that should make this easy to do, right? No. This is the basic code from tkinter import * window = Tk() window.geometry("200x200") btn1 = Button(window, text="btn1") btn1.grid(row=0, column=0, sticky="E") btn2 = Button(window, text="btn2") btn2.grid(row=0, column=1, sticky="E") window.mainloop()  What I want is: ------------------------- | ------ ------ | | |btn1| |btn2| | | ------ ------ | -------------------------  What I'm getting is: ------------------------- | ------ ------ | | |btn1| |btn2| | | ------ ------ | -------------------------  Which is sticking to the West even though I specified East What am I doing wrong? How can I get the result I want?

Answer: Give column 0 some weigth. It will force column 1 and 2 to right. from tkinter import * window = Tk() window.geometry("200x200") window.columnconfigure(0, weight=3) btn1 = Button(window, text="btn1") btn1.grid(row=0, column=1) btn2 = Button(window, text="btn2") btn2.grid(row=0, column=2) window.mainloop()  Another way to do this is using pack() and Frame() from tkinter import * window = Tk() window.geometry("200x200") cbframe = Frame(window) btn1 = Button(cbframe, text="btn1") btn1.pack(side="right", fill=None, expand=False) btn2 = Button(cbframe, text="btn2") btn2.pack(side="right", fill=None, expand=False) cbframe.pack(side="top", fill="both", expand=False) window.mainloop()
Question: I'm trying to install Matplotlib on a 32-bit version of Python. When I run pip install matplotlib, I get the following error when it tries to "Prepare metadata."  WARNING: Failed to activate VS environment: Could not find C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe ..\meson.build:1:0: ERROR: Unknown compiler(s): [['icl'], ['cl'], ['cc'], ['gcc'], ['clang'], ['clang-cl'], ['pgcc']] The following exception(s) were encountered: Running `icl ""` gave "[WinError 2] The system cannot find the file specified" Running `cl /?` gave "[WinError 2] The system cannot find the file specified" Running `cc --version` gave "[WinError 2] The system cannot find the file specified" Running `gcc --version` gave "[WinError 2] The system cannot find the file specified" Running `clang --version` gave "[WinError 2] The system cannot find the file specified" Running `clang-cl /?` gave "[WinError 2] The system cannot find the file specified" Running `pgcc --version` gave "[WinError 2] The system cannot find the file specified"  I really just want this to work on any 32-bit version of Python. This is on Python 3.9.1, but I've tried the most recent version of Python and I get the same error. I've tried different Matplotlib versions too. Really early Matplotlib versions (2.0.x,1.4.3) give me a different error, but it still doesn't work. Thanks for your help!

Answer: I had this same issue and here's how I solved it. Originally, I wanted to use Numpy, Scipy, Matplotlib, PyQt, and pyinstaller on a 32-bit version of Python. I've found out these libraries need to have "wheels" (I don't know what that actually is) that are compatible with 32-bit Python. That means that on pypi.org under "Download Files" of the specific library's version, there needs to be a file listed that goes like "library-version-pythoncompatibility-win32.whl". This means that it can be installed on a 32-bit version of python. Additionally, the libraries I listed aren't very robust in terms of 32-bit versions of python, so I needed to find versions of the above libraries that are all compatible with the same version of 32-bit Python. This is what I got to work. Python: 3.9.13 (32-bit) numpy: 1.23.5 scipy: 1.8.1 matplotlib: 3.7.5 PyQt5: 5.15.10 pyinstaller: 6.8.0 Each library version is compatible with more Python 32-bit versions than just 3.9.13, but 3.9.13 I found works for all of them at the same time. I am obviously not a computer scientist, so please add anything that might be more useful.
Question: I have this template: % template.tmpl file: % set result = fractions.Fraction(a*d + b*c, b*d) %} The solution of ${{a}}/{{b}} + {{c}}/{{d}}$ is ${{a * d + b*c}}/{{b*d}} = {{result.numerator}}/{{result.denominator}}$  which I invoke by from jinja2 import Template import fractions with open("c.jinja") as f: t = Template(f.read()) a = 2 b = 3 c = 4 d = 5 print(t.render(a = a, b = b, c=c, d=d))  I get jinja2.exceptions.UndefinedError: 'fractions' is undefined  but I want The solution of $2/3 + 4/5$ is $22/15=22/15$.  Is this possible to achieve that?

Answer: It can be done in two ways.  Calculate the result in Python code and then pass it as a parameter to the template. You can pass the fractions module as a parameter to the template.  Folder structure: . ├── c.jinja └── template_example.py  Option 1: Pass result to the template: template_example.py: from jinja2 import Template import fractions with open("c.jinja") as f: t = Template(f.read()) a = 2 b = 3 c = 4 d = 5 result = fractions.Fraction(a * d + b * c, b * d) print(t.render(a=a, b=b, c=c, d=d, result=result))  c.jinja: The solution of ${{a}}/{{b}} + {{c}}/{{d}}$ is ${{a * d + b*c}}/{{b*d}} = {{result.numerator}}/{{result.denominator}}$  Option 2: Pass the fractions: template_example.py: from jinja2 import Template import fractions with open("c.jinja") as f: t = Template(f.read()) a = 2 b = 3 c = 4 d = 5 print(t.render(a=a, b=b, c=c, d=d, fractions=fractions))  c.jinja: {% set result = fractions.Fraction(a*d + b*c, b*d) %} The solution of ${{a}}/{{b}} + {{c}}/{{d}}$ is ${{a * d + b*c}}/{{b*d}} = {{result.numerator}}/{{result.denominator}}$  Output for both options: The solution of $2/3 + 4/5$ is $22/15 = 22/15$  I would recommend Option 1: Pass the result to the template. This approach keeps the template focused on its primary role: rendering text with provided data, while the Python code handles all the computations.
Question: I have two tables which I need to join on a date column. I want to preserve all the dates in both tables, with the empty rows in each table just being filled with NaNs in the final combined array. I think an outer join is what I'm looking for. So I've written this code (with data_1 and data_2 acting as mockups of my actual tables) import pandas as pd def main(): data_1 = [["May-2024", 10, 5], ["June-2024", 3, 5], ["April-2015", 1, 3]] df1 = pd.DataFrame(data_1, columns = ["Date", "A", "B"]) df1["Date"] = pd.to_datetime(df1["Date"], format="%B-%Y") print(df1) data_2 = [["01-11-2024", 10, 5], ["01-06-2024", 3, 5], ["01-11-2015", 1, 3]] df2 = pd.DataFrame(data_2, columns = ["Date", "C", "D"]) df2["Date"] = pd.to_datetime(df2["Date"], format="%d-%m-%Y") print(df2) merged = df1.join(df2, how="outer", on=["Date"]) print(merged) if __name__ == "__main__": main()  But when I try and perform an outer join on two pandas dataframes, I get the error ValueError: You are trying to merge on object and int64 columns for key 'Date'. If you wish to proceed you should use pd.concat  I checked the datatype of both columns by printing print(df1["Date"].dtype, df2["Date"].dtype)  and they both seem to be datetime64[ns] datetime64[ns]  datetimes. So I'm not quite sure why I'm getting a ValueError Any help is appreciated, thanks.

Answer: You need to use merge, not join (that will use the index): # ensure datetime df1['Date'] = pd.to_datetime(df1['Date'], format='%B-%Y') df2['Date'] = pd.to_datetime(df2['Date'], dayfirst=True) # use merge merged = df1.merge(df2, how='outer', on=['Date'])  For join to work: merged_df = df1.set_index('Date').join(df2.set_index('Date')).reset_index()  Output:  Date A B C D 0 2015-04-01 1.0 3.0 NaN NaN 1 2015-11-01 NaN NaN 1.0 3.0 2 2024-05-01 10.0 5.0 NaN NaN 3 2024-06-01 3.0 5.0 3.0 5.0 4 2024-11-01 NaN NaN 10.0 5.0
Question: I want to show a figure that is calculated in a loop, let say with 5 iterations. This is the code that I wrote import numpy as np import matplotlib.pyplot as plt x = np.linspace(0,1,100) y = np.linspace(0,1,100) xx,yy = np.meshgrid(x,y) for n in range(5): a = np.sin(xx-2*n) plt.imshow(a,interpolation='bilinear') plt.show()  With this code, I got 5 figures. How to make it runs in one figure for each iteration? I used google collab, is it possible to make the result (figure) opened in new window (undocked) like in matlab?

Answer: You can simulate an animation by using a display/clear_output from ipython (used by Colab): import time import matplotlib.pyplot as plt import numpy as np from IPython.display import clear_output, display x = np.linspace(0, 1, 100) y = np.linspace(0, 1, 100) xx, yy = np.meshgrid(x, y) # you could initialize a subplots or whatever here.. for n in range(5): a = np.sin(xx - 2 * n) plt.imshow(a, interpolation="bilinear") # this one is optional (to verbose my output) plt.gca().set_title(f"Plot n°{n+1}", fontweight="bold") # added these three lines display(plt.gcf()) clear_output(wait=True) time.sleep(0.5) plt.show();  NB: This works in any IPython environment (e.g., Jupyter, Colab, ..).
Question: I am doing the following in spark sql: spark.sql(""" SELECT data.data.location.geometry.coordinates[0] FROM df""")  This works fine, however I do not want to use raw SQL, I use dataframe API like so: df.select("data.data.location.geometry.coordinates[0]")  Unfortunately this does not work: AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "data.data.location.geometry.coordinates[0]" due to data type mismatch: Parameter 2 requires the "INTEGRAL" type, however "0" has the type "STRING".; 'Project [data#680.data.location.geometry.coordinates[0] AS 0#697] +- Relation [data#680,id#681,idempotencykey#682,source#683,specversion#684,type#685] json  I know that I can use the F.col api and go with a getItem(0), but is there a built-in way to have the shortcut of getItem? '.' is the shortcut of getField is there one for array slicing? Thank you for your insight

Answer: TL;DR; use selectExpr() df.selectExpr('data.data.location.geometry.coordinates[0]')  Looks like indexing inside an array using [] is considered an expression, where as data.data.location... is considered a column name. DataFrame.select() takes column name as arguments, so doesn't understand [0], what you want is DataFrame.selectExpr(). >>> >>> df = spark.createDataFrame([({ ... "data": { ... "location": { ... "geometry": { ... "coordinates": [41.84201, -89.485937] ... } ... } ... } ... },)]).withColumnRenamed('_1', 'data') >>> df.show(truncate=False) +-----------------------------------------------------------------------------+ |data | +-----------------------------------------------------------------------------+ |{data -> {location -> {geometry -> {coordinates -> [41.84201, -89.485937]}}}}| +-----------------------------------------------------------------------------+ >>> df.printSchema() root |-- data: map (nullable = true) | |-- key: string | | |-- key: string | | |-- value: map (valueContainsNull = true) | | | |-- key: string | | | |-- value: map (valueContainsNull = true) | | | | |-- key: string | | | | |-- value: array (valueContainsNull = true) | | | | | |-- element: double (containsNull = true) >>>  >>> df.select('data.data.location.geometry.coordinates').show(truncate=False) +----------------------+ |coordinates | +----------------------+ |[41.84201, -89.485937]| +----------------------+ >>> df.select('data.data.location.geometry.coordinates[1]').show() +--------------+ |coordinates[1]| +--------------+ | null| +--------------+ >>>  >>> df.selectExpr('data.data.location.geometry.coordinates[1]').show() +----------------------------------------------+ |data[data][location][geometry][coordinates][1]| +----------------------------------------------+ | -89.485937| +----------------------------------------------+ >>> df.selectExpr( ... 'data.data.location.geometry.coordinates[0] as coord_lat', ... 'data.data.location.geometry.coordinates[1] as coord_long' ... ).show() +---------+----------+ |coord_lat|coord_long| +---------+----------+ | 41.84201|-89.485937| +---------+----------+ >>>
Question: Recently I have been working with dictionaries and I was to told to make dictionaries out of two lists like this: zipped = {key: value for key, value in zip(drinks, caffeine)}  Later I forgot how to do that and found a different way that seems simpler to me: zipped = dict(zip(drinks, caffeine))  Is there something worse about using this second example instead of the first?

Answer: Dictionary/List comprehensions are great flexible tools. You were taught this way so that the solution was more flexible and aligns with current coding practices. It's a "skill issue" as they say. There's nothing wrong with either. Always read company coding guidelines where possible.
Question: I'm having problems trying to detect edges in images corresponding to holes in a glass sample. Images look like this: images Single sample:  Each image contains part of a hole that was cut into a glass sample. Inspecting the images with my eyes, I can clearly see two regions, the glass and the hole. Sadly, all methods of trying to properly detect the edge haven't led to good results. The main reason why my attempts have failed, is the insufficient contrast between the glass and the hole, I believe. The hole is not cut through the entire thickness of the glass, leaving a glass bottom in the hole scattering light back into the camera thus making the contrast worse for me. Image processing things I've already tried:  blurring (gaussian, bilateral) sharpening adjusting contrast mean shift filtering adaptive thresholding canny edge detection sobel edge detection watershed segmentation  When taking the images I'm using an industrial camera with a ring light made of LEDs. The ring light can only be turned on or off, I can't adjust directions of the light or brightness. Taking images with various exposure times and analogue gains hasn't yielded much, since the contrast would stay the same throughout the measurements. Does anyone have an idea what steps I could take in order to properly detect the edges in the images? Be it image processing, programming or tips on how to take better pictures, any idea is appreciated! Here's an excerpt of my script: import cv2 image = cv2.imread( r'path/to/images') blurred = cv2.GaussianBlur(image, (7, 7), 9) gray = cv2.cvtColor(blurred, cv2.COLOR_BGR2GRAY) thresh = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 71, 5) edges = cv2.Canny(thresh, 100, 200) contours, _ = cv2.findContours( edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) output_image = image.copy() for contour in contours: cv2.drawContours(output_image, [contour], -1, (0, 255, 0), 2) cv2.imshow('Circle Edge', cv2.resize(output_image, (1000, 1000))) cv2.waitKey(0) cv2.destroyAllWindows()  with the detected edges painted in green on the original image (found in the images in this post) Thanks in advance! mentioned in the previous text already various image processing / manipulation steps taken

Answer: It looks to me that the variance/uniformity is significantly different in the two regions of the image, so maybe consider calculating the variance/standard-deviation within each 25x25 pixel block and normalising the result. I am doing it with ImageMagick here, because I am quicker with that, but you can do the same with OpenCV: magick YOURIMAGE.bmp -statistic standarddeviation 25x25 -normalize result.png    If you then flood fill the homogenous area, with some tolerance, you'll get:
Question: I would like to have a Python function cartesian_product which takes a dictionary of lists as input and returns as output a list containing as elements all possible dictionary which can be formed by taking from each list one element. Here would be an example: Calling cartesian_product({1: ['a', 'b'], 2: ['c', 'd'], 3: ['e', 'f']})  should return [ {1: 'a', 2: 'c', 3: 'e'}, {1: 'a', 2: 'c', 3: 'f'}, {1: 'a', 2: 'd', 3: 'e'}, {1: 'a', 2: 'd', 3: 'f'}, {1: 'b', 2: 'c', 3: 'e'}, {1: 'b', 2: 'c', 3: 'f'}, {1: 'b', 2: 'd', 3: 'e'}, {1: 'b', 2: 'd', 3: 'f'} ] 

Answer: This should do the trick: import itertools def cartesian_product(d): return [dict(zip(d, p)) for p in itertools.product(*d.values())]  Demo: >>> d = {1: ['a', 'b'], 2: ['c', 'd'], 3: ['e', 'f']} >>> from pprint import pp >>> pp(cartesian_product(d)) [{1: 'a', 2: 'c', 3: 'e'}, {1: 'a', 2: 'c', 3: 'f'}, {1: 'a', 2: 'd', 3: 'e'}, {1: 'a', 2: 'd', 3: 'f'}, {1: 'b', 2: 'c', 3: 'e'}, {1: 'b', 2: 'c', 3: 'f'}, {1: 'b', 2: 'd', 3: 'e'}, {1: 'b', 2: 'd', 3: 'f'}]