eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am exploring a change from pandas to polars. I like what I see. Currently, it is simple to get the data into Pandas. cf = Cashflow.objects.filter(acct=acct).values() df = pd.DataFrame(cf)  So I figured it would be a simple change - but this will not work for me. df = pl.DataFrame(cf)  What is the difference between using a Django query and putting the data inside Polars? Thank you.

Answer: You just need to check the input parameters of Polars and the output data type of the Django queryset. In Polars the pl.DataFrame() constructor expects a list of dictionaries or a list of other data structures. Also when you run Cashflow.objects.filter(acct=acct).values() in Django, the result is a queryset of dictionaries, where each dictionary represents a row of data. an example is provided below: <QuerySet [{'id': 1, 'name': 'Beatles Blog', 'tagline': 'All the latest Beatles news.'}]>  So the only thing you need to do is to convert the resulting queryset into a list before passing it to Polars: cf_list = list(Cashflow.objects.filter(account=acct).values()) df = pl.DataFrame(cf_list)
Question: I have the following Pandas DataFrame: import pandas as pd from collections import Counter print(sentences)  the output is (yes, the column name is 0):  0 0 A 1 B 2 C 3 D 4 EEE ... ... 462064467 FGH 462064468 QRS 462064469 EEE 462064470 VWXYZ 462064471 !!! [462064472 rows x 1 columns]  I have a custom function to check whether the content in the column 0 has a length > 1 or not (just an example): def is_more_than_one_character(t): if len(t) > 1: return True else: return False  And I apply the function like this: counter = Counter(sentences.apply(is_more_than_one_character))  I wish to count the occurrence of each string having a length > 1. Here is the example output of print(counter): [(EEE, 2), (FGH, 1), (QRS, 1), (!!!, 1)...]  but currently, the output is: [(False, 460686058), (True, 1378414)]  What did I miss? I think I am close. Thanks in advance.

Answer: You could filter with str.len and boolean indexing then pass to value_counts: out = sentences.loc[sentences[0].str.len()>1, 0].value_counts()  Or count everything, then filter the keys: out = sentences[0].value_counts() out = out[out.index.str.len()>1]  Output: 0 EEE 2 FGH 1 QRS 1 VWXYZ 1 !!! 1 Name: count, dtype: int64  If you really need to use your function and original approach: out = Counter(sentences.loc[sentences[0].apply(is_more_than_one_character), 0])  Or vectorizing the function: from numpy import vectorize @vectorize def is_more_than_one_character(t): if len(t) > 1: return True else: return False s = sentences[0] out = Counter(s[is_more_than_one_character(s)])  Or, actually, since you'll have to loop anyway, better use pure python: out = Counter(filter(is_more_than_one_character, sentences[0]))  Output: Counter({'EEE': 2, 'FGH': 1, 'QRS': 1, 'VWXYZ': 1, '!!!': 1})
Question: Decimal (i.e. non-prefixed) integers in Python seem to have fewer features than prefixed integers. If I do 1.real I get a SyntaxError: invalid decimal literal. However, if I do 0x1.real, then I get no error and 1 is the result. (Same for 0b1.real and 0o1.real, though in Python2 01.real gives a syntax error as).

Answer: It's because the 1. lead-in is being treated as a floating-point literal and r is not a valid decimal digit. Hex literals, of the form you show, are integers, so are not ambiguous in being treated as a possible floating point (there is no 0x1.1). If you use (1).real to specify that the literal is just the 1, it works fine. The following (annotated) transcript may help: >>> 0x1.real # Can only be integer, so works as expected. 1 >>> 1.real # Treated as floating point, "r" is invalid, File "<stdin>", line 1 1.real ^ SyntaxError: invalid syntax >>> (1).real # Explicitly remove ambiguity, works. 1 >>> 0x1.1 # No float hex literal in this form. File "<stdin>", line 1 0x1.1 ^ SyntaxError: invalid syntax   For completeness, the lexical tokens for numerics (integral and float) can be found in the Python docs: integer ::= decinteger | bininteger | octinteger | hexinteger decinteger ::= nonzerodigit (["_"] digit)* | "0"+ (["_"] "0")* bininteger ::= "0" ("b" | "B") (["_"] bindigit)+ octinteger ::= "0" ("o" | "O") (["_"] octdigit)+ hexinteger ::= "0" ("x" | "X") (["_"] hexdigit)+ nonzerodigit ::= "1"..."9" digit ::= "0"..."9" bindigit ::= "0" | "1" octdigit ::= "0"..."7" hexdigit ::= digit | "a"..."f" | "A"..."F" floatnumber ::= pointfloat | exponentfloat pointfloat ::= [digitpart] fraction | digitpart "." exponentfloat ::= (digitpart | pointfloat) exponent digitpart ::= digit (["_"] digit)* fraction ::= "." digitpart exponent ::= ("e" | "E") ["+" | "-"] digitpart  You can see there that the floats do not allow for hexadecimal prefixes, which is why it can safely assume the . in 0x1.real is not part of the literal value. That's not the case for 1.real, it assumes that the . is preceding a fractional part of a float. A clever-enough lexer could, of course, detect an invalid decimal digit immediately after the . and therefore assume it's a method to call on an integer. But that introduces complexity and may have other problematic edge cases.
Question: I am trying to find whether string(sentence) starts with numbering or alphabet bullets followed by dot(.) or space. I have regex like: r'^(\(\d|\[a-z]\))\s +'  and r"^(?:\(\d+\)|\\[a-z]\.)\s*"  I tried it on example strings: (a). this is bullet Not a bullet, (b) its bullet again I am so relaxed that its not bullet. (1) Bullet again.  But when I try matches = re.findall(pattern, text, re.M)  I get an empty list. How can I fix it?

Answer: Executable code in Python Solution import re content = """ (a). this is bullet Not a bullet, (b) its bullet again I am so relaxed that its not bullet. (1) Bullet again. """ patter_str = r'\((?:\w+)\)\.?\s.*' pattern = re.compile(patter_str) matches = pattern.findall(content) for item in matches: print(item)  Output (a). this is bullet (b) its bullet again (1) Bullet again.  EDIT If you want to rescue the text after the bullet you can use this pattern: patter_str = r'\((?:\w+)\)\.?\s(.*)'
Question: After updating VS code to v1.92, the Python extension consistently fails to launch, indefinitely showing a spinner next to “Reactivating terminals…” on the status bar. Selecting OUTPUT > Python reveals the error Failed to resolve env "/mnt/data-linux/miniconda3". Here’s the error trace: 2024-08-07 18:35:35.873 [error] sendStartupTelemetry() failed. s [Error]: Failed to resolve env "/mnt/data-linux/miniconda3" at ae (/home/user/.vscode-insiders/extensions/ms-python.python-2024.12.2-linux-x64/out/client/extension.js:2:1968174) at oe (/home/user/.vscode-insiders/extensions/ms-python.python-2024.12.2-linux-x64/out/client/extension.js:2:1966134) at Immediate.<anonymous> (/home/user/.vscode-insiders/extensions/ms-python.python-2024.12.2-linux-x64/out/client/extension.js:2:1962428) at processImmediate (node:internal/timers:478:21) { code: -4, data: undefined }  How do I fix this? Restarting worked, but that's not sustainable.

Answer: This appears to be a bug related to the new "native" Python locator. You can go back to the old working version by adding the following line to the user settings JSON (until the bug in the native locator is fixed): "python.locator": "js",  Note that this workaround pins you to the legacy version which is not something you'll want to have around forever so you might want to report your issue on Github at https://github.com/microsoft/vscode-python/issues. There've been many issues already filed and many solved but it's a work in progress. Example issues:  https://github.com/microsoft/vscode-python/issues/23922 https://github.com/microsoft/vscode-python/issues/23963 https://github.com/microsoft/vscode-python/issues/23956
Question: In Polars I can do: import polars as pl df = pl.DataFrame({'a': [1,2,3], 'b': [4, 5, 6]}) df.select(pl.sum_horizontal('a', 'b')) shape: (3, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 5 │ │ 7 │ │ 9 │ └─────┘  Is there a way to do this with DuckDB?

Answer: COLUMNS() can now be unpacked as of DuckDB 1.1.0 This does allow you to use the list_* functions e.g. duckdb.sql(""" from df select list_sum(list_value(*columns(*))) """)  ┌──────────────────────────────────┐ │ list_sum(list_value(df.a, df.b)) │ │ int128 │ ├──────────────────────────────────┤ │ 5 │ │ 7 │ │ 9 │ └──────────────────────────────────┘
Question: How do you disable caching when working with pythons functools? We have a boolean setting where you can disable the cache. This is for working locally. I thought something like this: @lru_cache(disable=settings.development_mode) But there is no setting like that. Am I missing something? How are people doing this?

Answer: If you want to disable caching conditionally while using functools.lru_cache, you need to manage this manually, You need a decorator that can conditionally apply lru_cache based on your settings I am using the docs as the guide docsOnFunctools Quotes from docs:  If maxsize is set to None, the LRU feature is disabled and the cache can grow without bound.If typed is set to true, function arguments of different types will be cached separately. If typed is false, the implementation will usually regard them as equivalent calls and only cache a single result. (Some types such as str and int may be cached separately even when typed is false.)  from functools import lru_cache, wraps def remove_lru_cache(maxsize=128, typed=False, enable_cache=True): def decorator(func): if enable_cache: return lru_cache(maxsize=maxsize, typed=typed)(func) return func return decorator  Then you can apply it to your code # settings your_settings = { 'dev_mode': True #set False to enable caching } @remove_lru_cache(maxsize=None, typed=False, enable_cache=not settings['dev_mode']) def add(nums: List[int]) -> int: return sum(nums)
Question: I'm subclassing pandas DataFrame in a project of mine. Most pandas operations preserve the subclass type, but df.groupby().agg() does not. Is this a bug? Is there a known workaround? import pandas as pd class MySeries(pd.Series): pass class MyDataFrame(pd.DataFrame): @property def _constructor(self): return MyDataFrame _constructor_sliced = MySeries MySeries._constructor_expanddim = MyDataFrame df = MyDataFrame({"a": reversed(range(10)), "b": list('aaaabbbccc')}) print(type(df.groupby("b").sum())) # <class '__main__.MyDataFrame'> print(type(df.groupby("b").agg({"a": "sum"}))) # <class 'pandas.core.frame.DataFrame'>  It looks like there was an issue (described here) that fixed subclassing for df.groupby, but as far as I can tell df.groupby().agg() was missed. I'm using pandas version 2.0.3.

Answer: It turns out that groupby().agg() combines Series to build a DataFrame, so the subclassed Series constructor needs to be properly defined. See this documentation. The following code runs with no errors: import pandas as pd class MySeries(pd.Series): @property def _constructor(self): return MySeries @property def _constructor_expanddim(self): return MyDataFrame class MyDataFrame(pd.DataFrame): @property def _constructor(self): return MyDataFrame @property def _constructor_sliced(self): return MySeries df = MyDataFrame({"a": reversed(range(10)), "b": list('aaaabbbccc')}) assert isinstance(df.groupby("b").agg({"a": "sum"}), MyDataFrame)
Question: I am trying to make a library related to RGB pixel data, but cannot seem to save the image data correctly.  That is my output image. This is my code: pixelmanager.py from PIL import Image import numpy as np class MakeImgFromPixelRGB: def createIMG(PixelArray: list, ImgName: str, SaveImgAsFile: bool): # Convert the pixels into an array using numpy array = np.array(PixelArray, dtype=np.uint8) if SaveImgAsFile == True: new_image = Image.fromarray(array) new_image.save(ImgName) class getPixelsFromIMG: def __init__(self, ImagePath): im = Image.open(ImagePath, 'r') width, height = im.size pixel_values = list(im.getdata()) self.output = pixel_values  test.py import pixelmanager a = pixelmanager.getPixelsFromIMG(ImagePath="download.jpg") pixelmanager.MakeImgFromPixelRGB.createIMG(a.output, "output.png", True) with open("output.txt", "w") as f: for s in a.output: f.write(str(s) + "," + "\n")  I have tried to de-scale the image in paint.net and also tried to mess with the uint size.

Answer: Mark Setchell was ALMOST correct. His code did help me get an image, but it was repeated 4 times in one. Mark's line of code had a switchup (with h as height and w as width): array = np.array(PixelArray, dtype=np.uint8).reshape(h,w,3)  This is my line of code: array = np.array(PixelArray, dtype=np.uint8).reshape(w, h, 3)
Question: I generated this on wordcloud.com using one of the "themes".  I'd like to be able to do this with the python wordcloud library, but so far all I can achieve is a single background color (so all black, not grey and black). Can anyone give me a hint on how to add the additional background color, using matlab or imshow? Here is my code: import numpy as np import matplotlib.pyplot as plt from pathlib import Path from PIL import Image from wordcloud import WordCloud tmp = "some text about Dynasty TV show" alexis_mask = np.array(Image.open('resources/alexis-poster-3.png')) print(repr(alexis_mask)) alexis_mask[alexis_mask == 0] = 255 def color_func(word, font_size, position, orientation, random_state=None, **kwargs): return "hsl(0, 100%, 27%)" wc = WordCloud(background_color="black", mask=alexis_mask, max_words=500,contour_width=2, contour_color="black") wc.generate(tmp) plt.figure(figsize=(28, 20)) plt.imshow(wc.recolor(color_func=color_func, random_state=3),interpolation="bilinear") plt.imshow(wc)  I've tried starting with a black/white image, and with a black/white/grey image. So far neither works. I don't think it's offered in the Wordcloud library but is it something I could do using imshow(), after I apply wordcloud? Thanks.

Answer: You can draw the wordcloud with the desired background color (e.g."grey") and then overlay this plot with a uniformly colored image (e.g. "black") masked using the wordcloud mask. import numpy as np import matplotlib.pyplot as plt from wordcloud import WordCloud from PIL import Image text = "some text about Dynasty TV show" x, y = np.ogrid[:300, :300] mask = (((x - 150)**2 + (y - 150)**2 > 130**2) * 255).astype(int) wc = WordCloud(background_color="grey", repeat=True, mask=mask) wc.generate(text) plt.axis("off") plt.imshow(wc) plt.imshow(np.dstack([Image.new("RGB", (wc.width,wc.height), "black"), wc.mask])) plt.show()   The minimal expample works for the case where the mask is 0 and 255 only. In the general case of a 1D mask it would be (wc.mask == 255) * 255, for a RGBA mask it would be ((wc.mask[:,:,0:3] == [255, 255, 255]) * 255)[:,:,0].
Question: I wanted to add file name pattern in AWS Glue ETL job python script where it should generate the files in s3 bucket with pattern dostrp*.csv.gz but could not find way how to provide this file pattern in python script : import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job args = getResolvedOptions(sys.argv, ['target_BucketName', 'JOB_NAME']) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args['JOB_NAME'], args) outputbucketname = args['target_BucketName'] # Script generated for node AWS Glue Data Catalog AWSGlueDataCatalog_node188777777 = glueContext.create_dynamic_frame.from_catalog(database="xxxx", table_name="xxxx", transformation_ctx="AWSGlueDataCatalog_node887777777") # Script generated for node Amazon S3 AmazonS3_node55566666 = glueContext.write_dynamic_frame.from_options(frame=AWSGlueDataCatalog_node8877777777, connection_type="s3", format="csv", format_options={"separator": "|"}, connection_options={"path": outputbucketname, "compression": "gzip", "partitionKeys": []}, transformation_ctx="AmazonS3_node5566677777") job.commit() 

Answer: Use pandas : import pandas as pd # Convert DynamicFrame to Pandas DataFrame df = dynamic_frame.toDF().toPandas() # Define S3 bucket and prefix s3_bucket = 'your-s3-bucket' s3_prefix = 'your/s3/prefix/' # Define the S3 path for the output file s3_output_path = f"s3://{s3_bucket}/{s3_prefix}output_file.csv.gz" # Create an in-memory buffer buffer = io.BytesIO() # Save the DataFrame as a CSV file in the buffer with gzip compression df.to_csv(buffer, index=False, compression='gzip') # Upload the buffer to S3 s3_client = boto3.client('s3') s3_client.put_object(Bucket=s3_bucket, Key=f"{s3_prefix}output_file.csv.gz", Body=buffer.getvalue())
Question: I'm trying to create an image using networkx, save that image to use later, and then overlay a plot over top of it later. However, when I try to load the image in and make new points, the scale seems off. I've tried everything I can find to make them sync, and I'm not sure what else to try at this point. Here's a simple example: import networkx as nx import matplotlib.pyplot as plt import numpy as np fig = plt.figure() G = nx.dodecahedral_graph() pos = nx.spring_layout(G) plt.box(False) nx.draw_networkx_edges(G, pos=pos) fig.canvas.draw() data = np.array(plt.gcf().canvas.get_renderer().buffer_rgba(), dtype=np.uint8) extent = list(plt.xlim() + plt.ylim())   So now I have a graph and have saved that image to data, and have saved the range of that graph to extent. I then want to replot that graph from data and overlay the nodes of the graph, in the positions stored in pos. plt.imshow(data, extent=extent) plt.box(False) nx.draw_networkx_nodes(G, pos=pos, node_color='green')   For some reason, the scale of the original image is shrunk, so the nodes end up being at a larger scale and not matching the edges. Is it something in the way I'm saving the image?

Answer: It seems that matplotlib adds padding to the sides of the image when saving the data from the plot. You can remove this padding by adding fig.tight_layout(pad=0) to the code like so: import networkx as nx import matplotlib.pyplot as plt import numpy as np fig = plt.figure() G = nx.dodecahedral_graph() pos = nx.spring_layout(G) plt.box(False) nx.draw_networkx_edges(G, pos=pos) fig.tight_layout(pad=0) fig.canvas.draw() data = np.array(plt.gcf().canvas.get_renderer().buffer_rgba(), dtype=np.uint8) extent = list(plt.xlim() + plt.ylim())
Question: The following data frame represents basic flatten tree structure, as shown below, where pairs (id, sub-id) and (sub-id, key) are always unique and key always represents the same thing under the same id id1 └─┬─ sub-id │ │ └─── key1 │ │ └─── value │ └─ sub-id2 │ └─── key1 │ └─── None id2 └─── sub-id3 └─── key2 └─── value  with graphical representation out of the way, below is the definition as polars.DataFrame df = pl.DataFrame( { "id": [1, 1, 2, 2, 2, 3], "sub-id": [1, 1, 2, 3, 3, 4], "key": ["key_1_1", "key_1_2", "key_2_1", "key_2_1", "key_2_2", "key_3"], "value": ["value1 1", "value 1 2", None, "value 2 1", "value 2 2", "value 3"], } )  The same data frame in table representation: shape: (6, 4) ┌────┬────────┬─────────┬───────────┐ │ ID │ sub-id │ key │ value │ ╞════╪════════╪═════════╪═══════════╡ │ 1 │ 1 │ key_1_1 │ value 1 │ │ 1 │ 1 │ key_1_2 │ value 2 │ │ 2 │ 2 │ key_2_1 │ value 2 1 │ │ 2 │ 3 │ key_2_1 │ None │ │ 2 │ 3 │ key_2_2 │ value 2 2 │ │ 3 │ 4 │ key_3 │ value 3 │ └────┴────────┴─────────┴───────────┘  How would I to fill gaps like shown below using polars. total size of data is about 100k rows. shape: (6, 4) ┌────┬────────┬─────────┬───────────┐ │ ID │ sub-id │ key │ value │ ╞════╪════════╪═════════╪═══════════╡ │ 1 │ 1 │ key_1_1 │ value 1 │ │ 1 │ 1 │ key_1_2 │ value 2 │ │ 2 │ 2 │ key_2_1 │ value 2 1 │ │ 2 │ 3 │ key_2_1 │ value 2 1 │ │ 2 │ 3 │ key_2_2 │ value 2 2 │ │ 3 │ 4 │ key_3 │ value 3 │ └────┴────────┴─────────┴───────────┘ 

Answer: There is pl.Expr.fill_null to fill missing values. As fill value, we use the first non-null value with the same id and key. As we assume that all values for the same id and key are the same, taking the first value is reasonable. It can be constructed as follows:  pl.Expr.filter and pl.Expr.is_not_null to filter for non-null values, pl.Expr.first to select the first such value, the window function pl.Expr.over to evaluate the expression separately for each id and key pair.  df.with_columns( pl.col("value").fill_null( pl.col("value").filter(pl.col("value").is_not_null()).first().over("id", "key") ) )  shape: (6, 4) ┌─────┬────────┬─────────┬───────────┐ │ id ┆ sub-id ┆ key ┆ value │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str ┆ str │ ╞═════╪════════╪═════════╪═══════════╡ │ 1 ┆ 1 ┆ key_1_1 ┆ value 1 │ │ 1 ┆ 1 ┆ key_1_2 ┆ value 2 │ │ 2 ┆ 2 ┆ key_2_1 ┆ value 2 1 │ │ 2 ┆ 3 ┆ key_2_1 ┆ value 2 1 │ │ 2 ┆ 3 ┆ key_2_2 ┆ value 2 2 │ │ 3 ┆ 4 ┆ key_3 ┆ value 3 │ └─────┴────────┴─────────┴───────────┘  Note. I needed to adapt your code slightly to match the example you described in the text. df = pl.DataFrame({ "id": [1, 1, 2, 2, 2, 3], "sub-id": [1, 1, 2, 3, 3, 4], "key": ["key_1_1", "key_1_2", "key_2_1", "key_2_1", "key_2_2", "key_3"], "value": ["value 1", "value 2", None, "value 2 1", "value 2 2", "value 3"], })
Question: I've got a string: s = ".,-2gg,,,-2gg,-2gg,,,-2gg,,,,,,,,t,-2gg,,,,,,-2gg,t,,-1gtt,,,,,,,,,-1gt,-3ggg"  and a regular expression I'm using import re delre = re.compile('-[0-9]+[ACGTNacgtn]+') #this is almost correct print (delre.findall(s))  This returns: ['-2gg', '-2gg', '-2gg', '-2gg', '-2gg', '-2gg', '-1gtt', '-1gt', '-3ggg']  But -1gtt and -1gt are not desired matches. The integer in this case defines how many subsequent characters to match, so the desired output for those two matches would be -1g and -1g, respectively. Is there a way to grab the integer after the dash and dynamically define the regex so that it matches that many and only that many subsequent characters?

Answer: You can't do this with the regex pattern directly, but you can use capture groups to separate the integer and character portions of the match, and then trim the character portion to the appropriate length. import re # surround [0-9]+ and [ACGTNacgtn]+ in parentheses to create two capture groups delre = re.compile('-([0-9]+)([ACGTNacgtn]+)') s = ".,-2gg,,,-2gg,-2gg,,,-2gg,,,,,,,,t,-2gg,,,,,,-2gg,t,,-1gtt,,,,,,,,,-1gt,-3ggg" # each match should be a tuple of (number, letter(s)), e.g. ('1', 'gtt') or ('2', 'gg') for number, bases in delre.findall(s): # print the number, then use slicing to truncate the string portion print(f'-{number}{bases[:int(number)]}')  This prints -2gg -2gg -2gg -2gg -2gg -2gg -1g -1g -3ggg  You'll more than likely want to do something other than print, but you can format the matched strings however you need! NOTE: this does fail in cases where the integer is followed by fewer matching characters than it specifies, e.g. -10agcta is still a match even though it only contains 5 characters.
Question: Currently I have a bunch of code that does this: from typing import Dict foo: Dict[str, str] = []  In Python 3.9+, it is preferable to use the built-in types (source): foo: dict[str, str] = []  Is there a way to configure mypy to raise an error/warning when my code uses Dict instead of dict?

Answer: According to the mypy maintainers, this fuctionality will not be implemented because it has already been implemented by formatters like ruff:  This is already implemented by linters such as Ruff (https://docs.astral.sh/ruff/rules/non-pep585-annotation/). That doesn't mean mypy can't also implement support for checks like this, but it does mean the return on investment for mypy to add this feature is lower.  Source