eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Yes, I'm, of the understanding that black gives very little leeway in getting it to act differently but I was wondering about the best way to handle something like this (my original code): @dataclass class Thing1: property1: int # The first property. property2: typing.List[int] # This is the second property # and the comment crosses multiple lines.  Now, when I run that through black, it gives me: @dataclass class Thing1: property1: int # The first property. property2: typing.List[int] # This is the second property # and the comment crosses multiple lines.  which is not really laid out as I'd like. Is there a way to get black to keep the comments lined up? I don't care which column it starts in per field, but it would be nice to return to the readable nature, with the comments within each field being lined up: @dataclass class Thing1: property1: int # The first property. property2: typing.List[int] # This is the second property # and the comment crosses multiple lines.  If there's no way to do this, I'll probably either place multi-line comments before each field, or ensure all comments fit on the one line and add lengthier explanatory notes at the end of the dataclass definition: @dataclass class Thing1: property1: int # The first property. # This is the second property and the comment, while it can # have more characters, still crosses multiple lines. property2: typing.List[int] @dataclass class Thing2: property1: int # The first property. property2: typing.List[int] # This is the second property. # Note that the comments above now fit on the same line as the # field, and this comment here is meant to provide any needed # multi-line detail for ALL fields in this class.  That at least is still somewhat local to the data fields. But I'd rather find a way to have it line up comments similar to what I originally had, if possible. Happy to entertain any suggestions.

Answer: You can wrap your block with # fmt: on/off, so Black doesn't touch it. # fmt: off @dataclass class Thing1: property1: int # The first property. property2: typing.List[int] # This is the second property # and the comment crosses multiple lines. # fmt: on  I usually prefer to relocate the comments and stick with default Black formatting.
Question: Using Napari Image Analysis GUI to run the Allen Cell Segmenter (no response in Napari github or Allen Cell Forum, thought I'd try here) and getting the following error when I attempt to run the watershed for cutting function: ImportError: cannot import name 'watershed' from 'skimage.morphology' (C:\Users\Murryadmin\anaconda3\envs\napari-env\lib\site-packages\skimage\morphology_init_.py)  c:\users\murryadmin\anaconda3\envs\napari-env\lib\site-packages\aicssegmentation\core\utils.py(449)watershed_wrapper() -> from skimage.morphology import watershed, dilation, ball  Anyone have any potential fixes for this? Thanks

Answer: watershed was moved from skimage.morphology to skimage.segmentation in version 0.17. There was a pointer from morphology to the new function in segmentation in 0.17 and 0.18, but it was removed in 0.19. The Allen Cell Segmenter needs to be updated to match the more modern scikit-image version, so I would raise an issue in their GitHub repository if I were you. Downgrading to scikit-image 0.18 could fix the Allen Cell Segmenter itself, but unfortunately napari requires 0.19+.
Question:   I keep on getting these messages on the bottom right corner of my screen when opening up VS code. Any idea on how to get rid of it? I can still write code and run the code fine but I don't understand why this is happening. I've tried by deleting the python extension and anything related to python in the video extensions tab, but no luck.

Answer: Please update the python extension to the latest version  or install the pre-release version directly. ( may be more useful to you )  this will basically solve your problem. If the error continues, follow these steps:   Uninstall Python extension (if you have pylance uninstall it first). Uninstall any other extension that is failing. Close all instances of VS Code. Go to, %USERPROFILE%/.vscode/extensions (on windows) or ~/.vscode/extensions on Linux/Mac. Delete any folder with the name starting with ms-python.python* Start VS Code, and install Python extension (also pylance if you uninstalled it in step 1).
Question: I have a PySpark dataframe which looks like this: Id timestamp col1 col2 abc 789 0 1 def 456 1 0 abc 123 1 0 def 321 0 1  I want to group by or partition by ID column and then the lists for col1 and col2 should be created based on the order of timestamp. Id timestamp col1 col2 abc [123,789] [1,0] [0,1] def [321,456] [0,1] [1,0]  My approach: from pyspark.sql import functions as F from pyspark.sql import Window as W window_spec = W.partitionBy("id").orderBy('timestamp') ranged_spec = window_spec.rowsBetween(W.unboundedPreceding, W.unboundedFollowing) df1 = df.withColumn("col1", F.collect_list("reco").over(window_spec))\ .withColumn("col2", F.collect_list("score").over(window_spec))\ df1.show()  But this is not returning list of col1 and col2.

Answer: I don't think the order can be reliably preserved using groupBy aggregations. So window functions seems to be the way to go. Setup: from pyspark.sql import functions as F, Window as W df = spark.createDataFrame( [('abc', 789, 0, 1), ('def', 456, 1, 0), ('abc', 123, 1, 0), ('def', 321, 0, 1)], ['Id', 'timestamp', 'col1', 'col2'])  Script: w1 = W.partitionBy('Id').orderBy('timestamp') w2 = W.partitionBy('Id').orderBy(F.desc('timestamp')) df = df.select( 'Id', *[F.collect_list(c).over(w1).alias(c) for c in df.columns if c != 'Id'] ) df = (df .withColumn('_rn', F.row_number().over(w2)) .filter('_rn=1') .drop('_rn') )  Result: df.show() # +---+----------+------+------+ # | Id| timestamp| col1| col2| # +---+----------+------+------+ # |abc|[123, 789]|[1, 0]|[0, 1]| # |def|[321, 456]|[0, 1]|[1, 0]| # +---+----------+------+------+   You were also very close to what you needed. I've played around and this seems to be working too: window_spec = W.partitionBy("Id").orderBy('timestamp') ranged_spec = window_spec.rowsBetween(W.unboundedPreceding, W.unboundedFollowing) df1 = (df .withColumn("timestamp", F.collect_list("timestamp").over(ranged_spec)) .withColumn("col1", F.collect_list("col1").over(ranged_spec)) .withColumn("col2", F.collect_list("col2").over(ranged_spec)) ).drop_duplicates() df1.show()
Question: Using the threshold functions in open CV on an image to get a binary image, with Otsu's thresholding I get a image that has white spots due to different lighting conditions in parts of the image or with adaptive threshold to fix the lighting conditions, it fails to accurately represent the pencil-filled bubbles that Otsu actually can represent.  How can I get both the filled bubbles represented and a fixed lighting conditions without patches? Here's the original image  Here is my code  #binary image conversion thresh2 = cv2.adaptiveThreshold(papergray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 21, 13) thresh = cv2.threshold(papergray, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1] cv2.imshow("Binary", thresh) #Otsu's cv2.imshow("Adpative",thresh2) 

Answer: Problems with your approach: The methods you have tried out:  Otsu threshold is decided based on all the pixel values in the entire image (global technique). If you look at the bottom-left of your image, there is a gray shade which can have an adverse effect in deciding the threshold value. Adaptive threshold: here is a recent answer on why it isn't helpful. In short, it acts like an edge detector for smaller kernel sizes  What you can try: OpenCV's ximgproc module has specialized binary image generation methods. One such method is the popular Niblack threshold technique. This is a local threshold technique that depends on statistical measures. It divides the image into blocks (sub-images) of size predefined by the user. A threshold is set based on the mean minus k times standard deviation of pixel values for each block. The k is decided by the user. Code: img =cv2.imread('omr_sheet.jpg') blur = cv2.GaussianBlur(img, (3, 3), 0) gray = cv2.cvtColor(blur, cv2.COLOR_BGR2GRAY) niblack = cv2.ximgproc.niBlackThreshold(gray, 255, cv2.THRESH_BINARY, 41, -0.1, binarizationMethod=cv2.ximgproc.BINARIZATION_NICK)  Result:  Links:  To know more about cv2.ximgproc.niBlackThreshold  There are other binarization techniques available that you may want to explore. It also contains links to research papers that explain each of these techniques on detail.   Edit: Adaptive threshold actually works if you know what you are working with. You can decide the kernel size beforehand. See Prashant's answer.
Question: I'm getting this error: At line:1 char:1 + playwright install + ~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (playwright:String) [],CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException  I'm installing it using pip, for use in python

Answer: You need playwright added to your PATH. However, a better way to do this (if python is added to your path) without adding it to your PATH is by running: python -m playwright install  This runs the playwright module as a script. Use python -h for more information on these flags, and python -m playwright for more information on the flags supported specifically by playwright
Question: This question is probably me not understanding architecture of (new) sqlalchemy, typically I use code like this: query = select(models.Organization).where( models.Organization.organization_id == organization_id ) result = await self.session.execute(query) return result.scalars().all()  Works fine, I get a list of models (if any). With a query with specific columns only: query = ( select( models.Payment.organization_id, models.Payment.id, models.Payment.payment_type, ) .where( models.Payment.is_cleared.is_(True), ) .limit(10) ) result = await self.session.execute(query) return result.scalars().all()  I am getting first row, first column only. Same it seems to: https://docs.sqlalchemy.org/en/14/core/connections.html?highlight=scalar#sqlalchemy.engine.Result.scalar My understanding so far was that in new sqlalchemy we should always call scalars() on the query, as described here: https://docs.sqlalchemy.org/en/14/changelog/migration_20.html#migration-orm-usage But with specific columns, it seems we cannot use scalars() at all. What is even more confusing is that result.scalars() returns sqlalchemy.engine.result.ScalarResult that has fetchmany(), fechall() among other methods that I am unable to iterate in any meaningful way. My question is, what do I not understand?

Answer:  My understanding so far was that in new sqlalchemy we should always call scalars() on the query  That is mostly true, but only for queries that return whole ORM objects. Just a regular .execute()  query = select(Payment) results = sess.execute(query).all() print(results) # [(Payment(id=1),), (Payment(id=2),)] print(type(results[0])) # <class 'sqlalchemy.engine.row.Row'>  returns a list of Row objects, each containing a single ORM object. Users found that awkward since they needed to unpack the ORM object from the Row object. So .scalars() is now recommended  results = sess.scalars(query).all() print(results) # [Payment(id=1), Payment(id=2)] print(type(results[0])) # <class '__main__.Payment'>  However, for queries that return individual attributes (columns) we don't want to use .scalars() because that will just give us one column from each row, normally the first column  query = select( Payment.id, Payment.organization_id, Payment.payment_type, ) results = sess.scalars(query).all() print(results) # [1, 2]  Instead, we want to use a regular .execute() so we can see all the columns  results = sess.execute(query).all() print(results) # [(1, 123, None), (2, 234, None)]  Notes:  .scalars() is doing the same thing in both cases: return a list containing a single (scalar) value from each row (default is index=0).  sess.scalars() is the preferred construct. It is simply shorthand for sess.execute().scalars().
Question: Here's my code using Python (simplified version): import yt_dlp YDL_OPTIONS = { 'format': 'bestaudio*', 'noplaylist': True, } with yt_dlp.YoutubeDL(YDL_OPTIONS) as ydl: info = ydl.extract_info(url, download=False)  The problem comes up when the url directs to a playlist (e.g. https://www.youtube.com/playlist?list=PLlrATfBNZ98dudnM48yfGUldqGD0S4FFb) Here's the output: [youtube:tab] PLlrATfBNZ98dudnM48yfGUldqGD0S4FFb: Downloading webpage [youtube:tab] PLlrATfBNZ98dudnM48yfGUldqGD0S4FFb: Downloading API JSON with unavailable videos [download] Downloading playlist: C++ [youtube:tab] playlist C++: Downloading 100 videos [download] Downloading video 1 of 100 [youtube] 18c3MTX0PK0: Downloading webpage [youtube] 18c3MTX0PK0: Downloading android player API JSON [download] Downloading video 2 of 100 [youtube] 1OsGXuNA5cc: Downloading webpage [youtube] 1OsGXuNA5cc: Downloading android player API JSON [download] Downloading video 3 of 100 [youtube] 1E_kBSka_ec: Downloading webpage [youtube] 1E_kBSka_ec: Downloading android player API JSON ...  As you can see, the 'noplaylist' option didn't work in this case. Is there an option or function making ydl only extract one video info, for instance the first one, in the entire playlist?

Answer:  noplaylist: Download single video instead of a playlist if in doubt.  Source: github.com/yt-dlp/yt-dlp What you shared with us isn't a playlist of a single video so, according to me, there isn't any doubt here. You can request yt_dlp to treat only the first video of the playlist by adding 'playlist_items': '1' to YDL_OPTIONS. Then the info concerning the video are in info['entries'][0].
Question: I'm trying to find a way to generate all possible "patterns" of length N out of a list of K letters. I've looked at similar questions but they all seem to be asking about combinations, permutations, etc. which is not exactly what I'm after. For example, let K = 3 and N = 2. That is, I want all 2-letter "patterns" that can be made with the letters [A, B, C]. AA is one such pattern. AB is another. And those are the only two. BB and CC are the same as AA, it's just "a letter, and then the same letter." Similarly, BA, BC, AC, etc. are the same as AB, it's just "a letter, and then a different letter." So for this simple case, there are only two patterns, and in fact this illustrates why K must be less than or equal to N (adding additional letters to choose from doesn't change anything). If instead, K = 3, N = 3, then the five possible patterns would be AAA, AAB, ABA, ABB, and ABC. Every other permutation of three letters has a pattern that is identical to one of those five. If K = 2 and N = 3, then there are just four possible patterns: AAA, AAB, ABA, ABB. (ABC is no longer a valid choice because I only have two letters to choose from.) Of course, these examples are trivial to do by hand - I'm trying to create code that will generate all possible patterns for larger values of N and K. This may be more of a pure mathematical question but ultimately I need a Python function that will produce these so I thought I'd try here first to see if anyone knows or can think of an efficient way to do this.

Answer: One of the comments, from @JacobRR, was very close to what we need. Each "pattern" actually corresponds to partitioning of set {1, 2, ..., N} into K subsets. Each subset (even empty!) corresponds to the positions where letter l_k should be placed (l_1 = A, l_2 = B etc.). There's a demo here. https://thewebdev.info/2021/10/28/how-to-generate-set-partitions-in-python For example, in case K=3, N=3, the partitions would be {1,2,3}, ∅, ∅ {1}, {2, 3}, ∅ {2}, {1, 3}, ∅ {3}, {1, 2}, ∅ {1}, {2}, {3}  and for K=2, N=3, it's {1,2,3}, ∅ {1}, {2, 3} {2}, {1, 3} {3}, {1, 2}  corresponding exactly to the given examples. This question is also relevant. https://codereview.stackexchange.com/questions/1526/finding-all-k-subset-partitions I also wrote my own naive implementation. import copy N = 3 K = 2 iter = min(N, K) def partitions(S, K): if K == 1: return [[S]] if len(S) == 0: return [[]] result = [] S_new = copy.copy(S) first = S_new.pop(0) if (K-1 <= len(S_new)): p1 = partitions(S_new, K-1) for p in p1: p.append([first]) result.append(p) if (K <= len(S_new)): p2 = partitions(S_new, K) for p in p2: for idx in range(len(p)): p_new = copy.deepcopy(p) p_new[idx].append(first) result.append(p_new) return result for idx in range(1, iter+1): print(partitions([i for i in range(0, N)], idx))
Question: I got an error when install dependencies for my project! OS : WinDow 11 Python: 3.10.4 (64bit) Pip: 22.1.2  Building wheel for django-admin-sortable2 (setup.py) ... done Created wheel for django-admin-sortable2: filename=django_admin_sortable2-0.7.5-py3-none-any.whl size=69989 sha256=0a4ff29d0c9b0422611dde61c6c1665dd36b10f98413f4ed7b8532e29afdc03d Stored in directory: c:\users\kev\appdata\local\pip\cache\wheels\99\3e\95\384eeaa2d641ef0c9e8b46e701737b53ae6a973358887816e0 Building wheel for easy-thumbnails (setup.py) ... done Created wheel for easy-thumbnails: filename=easy_thumbnails-2.7-py2.py3-none-any.whl size=69700 sha256=ce66afcd2ca403acf9225b53eed60300c8d37c3bad53dcdf37ebc3a25550bdc6 Stored in directory: c:\users\kev\appdata\local\pip\cache\wheels\cb\33\00\f7fa4b381ae4cbaf99674fb7a4411339d38e616cfcc41632c5 Building wheel for gevent (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for gevent (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [288 lines of output] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-cpython-310 creating build\lib.win-amd64-cpython-310\gevent copying src\gevent\ares.py -> build\lib.win-amd64-cpython-310\gevent copying src\gevent\backdoor.py -> build\lib.win-amd64-cpython-310\gevent copying src\gevent\baseserver.py -> build\lib.win-amd64-cpython-310\gevent copying src\gevent\builtins.py -> build\lib.win-amd64-cpython-310\gevent 

Answer: tl;dr: use python3.8 or update requirement.txt versions. More info: The combination of (A) gevent==20.9, (B) windows 10, and (C) python3.10 does not have a prebuilt wheel. You can check this kind of stuff by going to pypi and looking what is offered for downloads (https://pypi.org/project/gevent/20.9.0/#files) I'm assuming that you will not be able to compile things from source yourself (it's a hassle), so you need to change (A), (B) or (C). (A). Changing this means relaxing or updating the version requirements. For example gevent==21.12 does have a wheel for windows and python3.10 (B). Changing this means not using windows, probably not an option (C). Changing this means using older python version. For example, python3.8 has a wheel for gevent==20.9.
Question: Could you please explain how it works. In particular: where do a,c,n,s,z come from? And how to convert this to regular functions and their calls? x=10 y=5 abs( (lambda a: lambda z, c, n: a(a, z, c, n)) (lambda s, z, c, n: z if n == 0 else s(s, z*z+c, c, n-1)) (0, 0.02*x+0.05j*y, 10) )  It was even longer, but I figured out the rest of it. print('\n'.join([''.join(['*'if abs((lambda a: lambda z, c, n: a(a, z, c, n))(lambda s, z, c, n: z if n == 0 else s(s, z*z+c, c, n-1))(0, 0.02*x+0.05j*y, 40)) < 2 else ' ' for x in range(-80, 20)]) for y in range(-20, 20)]))  This monster expression outputs a pseudo-graphic Mandelbrot fractal to the console.

Answer: This function  (lambda a: lambda z, c, n: a(a, z, c, n)) can be rewritten as def apply_function_to_itself(funk): def inner(arg1, arg2, arg3): funk(funk, arg1, arg2, arg3) return inner  This is a functional thing that calls a function with itself as the first argument. This is necessary for recursion in functional languages but in python there is usually a better way. This function (lambda s, z, c, n: z if n == 0 else s(s, z*z+c, c, n-1)) can be rewritten as this: def transform_n_times(funk, value, c, n): if n==0: return arg1 else: return funk(funk, value * value + constant, constant, n-1)  We know from the previous function that funk === call_if_n_is_not_0, so this is a recursive function that will apply a operation n times before stopping. So basically combined we can rewrite it like this: def transform_n_times(value, constant, n): for i in range(n): value arg1 * arg1 + arg2 return value  Much simpler. Understanding the entire program The entire code could be written like this:  def apply_function_to_itself(funk): def inner(arg1, arg2, arg3): funk(funk, arg1, arg2, arg3) return inner def transform_n_times(funk, value, constant, n): if n==0: return arg1 else: return funk(funk, value * value + constant, constant, n-1) abs(apply_function_to_itself(call_if_n_is_not_0)(0, 0.02*x+0.05j*y, 10))  Or like this. We can get rid of the apply_function_to_itself entirely but keep the recursion like this: def transform_n_times(value, constant, n): if n==0: return arg1 else: return transform_n_times(funk, value * value + constant, constant, n-1) abs(transform_n_times(0, 0.02*x+0.05j*y, 10))  Or we can delete the recursion entirely and just use a loop: constant = 0.02*x+0.05j*y value = 0 n = 10 for i in range(n): a1 = value * value + constant abs(value)  The basic equation behind the mandlebrot set is f(x) = x^2+c which matches this.
Question: I remember I was able to reconnect to an existing Jupyter Notebook session in VS code before by changing the kernel for a notebook. Now the option to reconnect to an existing session is gone, see:  How do I reconnect to an existing Jupyter Notebook session in VS code? To be clear, the sessions were never shut down. In fact, I can still see them running in the Running tab of my browser version of Jupyter Notebook, although clicking on them results in a 404 error:  The Jupyter kernel is running on a remote server. I use an SSH session to connect back to it when I work. Current versions: VS code is v1.68.1 and the Jupyter extension on the remote machine is v2022.5.1001601848, if that helps.

Answer: If you would like to connect to an existing jupyter server can you do so by going into the command Jupyter: Specify Jupyter Server for Connections and selecting the appropriate remote server from the list. If the required server is not on that list, then you can select it.
Question: I have a list of items and want to clean the data with certain conditions and the output is a dataframe. Here's the list: [ "Onion per Pack|500 g|Rp18,100|Rp3,700 / 100 g|Add to cart", "Shallot per Pack|250 g|-|49%|Rp22,300|Rp11,300|Rp4,600 / 100 g|Add to cart", "Spring Onion per Pack|250 g|Rp7,000|Rp2,800 / 100 g|Add to cart", "Green Beans per Pack|250 g|Rp5,900|Rp2,400 / 100 g|Add to cart", ]  into     name unit discount price unit price     Onion per Pack 500 g  Rp18,100 Rp3,700 / 100 g   Shallot per Pack 250 g 49% Rp22,300 Rp11,300   Spring Onion per Pack 250 g  Rp7,000 Rp2,800 / 100 g   Green Beans per Pack 250 g  Rp5,900 Rp2,400 / 100 g     Currently my code is: datas = pd.DataFrame() for i in item: long = len(i.split("|")) if long == 5: data = {"name": i.split("|")[0] "unit": i.split("|")[2] "discount": "" "price": i.split("|")[3] "unit price": i.split("|")[4]} dat = pd.DataFrame(data) datas.append(dat) else: data = {"name": i.split("|")[0] "unit": i.split("|")[2] "discount": i.split("|")[4] "price": i.split("|")[6] "unit price": i.split("|")[7]} dat = pd.DataFrame(data) datas.append(dat)  Is there a more efficient way? A shorter way to achieve this?

Answer: I'd using a map; try: ids = { 5: [0, 2, -1, 3, 4], 8: [0, 2, 4, 6, 7] } datas = pd.DataFrame() for i in item: i = i.split("|") long = len(i) data = { "name": i[ids[long][0]], "unit": i[ids[long][1]], "discount": i[ids[long][2]] if ids[long][2] != -1 else "", "price": i[ids[long][3]], "unit price": i[ids[long][4]], } datas.append(pd.DataFrame(data))