eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to use Spacy to generate embeddings of text stored in a polars DataFrame and store the results in the same DataFrame. Next, I want to save this DataFrame to the disk and be able to load again as a polars DataFrame. The backtransformation from pandas to polars results in an error. This is the error message: ArrowInvalid: Could not convert Hello with type spacy.tokens.doc.Doc: did not recognize Python value type when inferring an Arrow data type Here is my code: from io import StringIO import polars as pl import pandas as pd import spacy nlp = spacy.load("de_core_news_sm") json_str = '[{"foo":"Hello","bar":6},{"foo":"What a lovely day","bar":7},{"foo":"Nice to meet you","bar":8}]' #Initalize and store DataFrame df = pl.read_json(StringIO(json_str)) df = df.with_columns(pl.col("foo").map_elements(lambda x: nlp(x)).alias("encoding")) df.to_pandas().to_pickle('pickled_df.pkl') #Load DataFrame df_loaded_pd = pd.read_pickle('pickled_df.pkl') df_loaded_pl = pl.from_pandas(df_loaded_pd)  These are the package versions I used: # Name Version Build Channel pandas 2.2.3 py312hf9745cd_1 conda-forge polars 1.9.0 py312hfe7c9be_0 conda-forge spacy 3.7.2 py312h6db74b5_0 spacy-curated-transformers 0.2.2 pypi_0 pypi spacy-legacy 3.0.12 pyhd8ed1ab_0 conda-forge spacy-loggers 1.0.5 pyhd8ed1ab_0 conda-forge  Thank you for your help!

Answer: Serializing and deserializing SpaCy objects within a polars DataFrame can be stored by using SpaCys native DocBin class. The following code generates doc objects, saves them locally, and successfully loads them afterwards. from io import StringIO from spacy.tokens import DocBin import polars as pl import spacy nlp = spacy.load("de_core_news_md") json_str = '[{"foo":"Hello","bar":6},{"foo":"What a lovely day","bar":7},{"foo":"Nice to meet you","bar":8}]' doc = nlp("some text") #Serialize polars DataFrame df = pl.read_json(StringIO(json_str)) df = df.with_columns(pl.col("foo").map_elements(lambda x: DocBin(docs=[nlp(x)]).to_bytes()).alias('binary_embbeding')) df.write_parquet('saved.pq') #Deserialize polars DataFrame df_loaded = pl.read_parquet('saved.pq') df_loaded = df_loaded.with_columns(pl.col('binary_embbeding').map_elements(lambda x: list(DocBin().from_bytes(x).get_docs(nlp.vocab))[0]).alias("spacy_embedding")) #Calculate similarity df_loaded.with_columns(pl.col("spacy_embedding").map_elements(lambda x: doc.similarity(x), return_dtype=pl.Float64).alias('Score'))  Applying functions to deserialized SpaCy objects Serializing and deserializing SpaCys objects with native polars functions (such as df.write_parquet()) heavily depends on the used model. In the above case the similarity calculation only works when utilizing SpaCys language model that contain wordvectors. nlp = spacy.load("de_core_news_sm") # Line 20 does not works nlp = spacy.load("de_core_news_md") # Line 20 works nlp = spacy.load("de_core_news_lg") # Line 20 works nlp = spacy.load("de_dep_news_trf") # Line 20 does not works
Question: Here's a small example: import altair as alt import polars as pl source = pl.DataFrame( { "Category": list("AAABBBCCC"), "Value": [0.1, 0.6, 0.9, 0.7, 0.2, 1.1, 0.6, 0.1, 0.2], "Date": [f"2024-{m+1}-1" for m in range(3)] * 3, } ).with_columns(pl.col("Date").str.to_date()) bars = alt.Chart(source).mark_bar().encode( x=alt.X("Date:T"), xOffset="Category:N", y="Value:Q", color="Category:N", ) bars   If I set x="Date:N", then the example behaves as I'd like, but without the benefits of temporal formatting for the x-axis:  Is there any way in which I can have xOffset work for the case where x="Date:T"?

Answer: If you use the ordinal or nominal data type, you can supply a timeUnit to get date formatting. There are many options depending on what kind of data you are working with. import altair as alt import polars as pl source = pl.DataFrame( { "Category": list("AAABBBCCC"), "Value": [0.1, 0.6, 0.9, 0.7, 0.2, 1.1, 0.6, 0.1, 0.2], "Date": [f"2024-{m+1}-1" for m in range(3)] * 3, } ).with_columns(pl.col("Date").str.to_date()) bars = alt.Chart(source.to_pandas()).mark_bar().encode( x=alt.X("Date:O", timeUnit="yearmonthdate"), xOffset="Category:N", y="Value:Q", color="Category:N", ) bars
Question: I am trying to write a function gaussian_elim which takes in an n x n numpy array A and an n x 1 numpy array b and performs Gaussian elimination on the augmented matrix [A|b]. It should return an n x (n+1) matrix of the form M = [U|c], where U is an n x n upper triangular matrix. However, when I test my code on a simple 2x2 matrix, it seems that the elimination step is not being performed properly. I have inserted print statements to illustrate how the matrix is not being updated properly. def gaussian_elim(A,b): """ A: n x n numpy array b: n x 1 numpy array Applies Gaussian Elimination to the system Ax = b. Returns a matrix of the form M = [U|c], where U is upper triangular. """ n = len(b) b = b.reshape(-1, 1) # Ensure b is a column vector of shape (n, 1) M = np.hstack((A,b)) #Form the n x (n+1) augmented matrix M := [A|b] #For each pivot: for j in range(n-1): #j = 0,1,...,n-2 #For each row under the pivot: for i in range(j+1,n): #i = j + 1, j + 2,..., n-1 if (M[j,j] == 0): print("Error! Zero pivot encountered!") return #The multiplier for the the i-th row m = M[i,j] / M[j,j] print("M[i,:] = ", M[i,:]) print("M[j,:] = ", M[j,:]) print("m = ", m) print("M[i,:] - m*M[j,:] = ", M[i,:] - m*M[j,:]) #Eliminate entry M[i,j] (the first nonzero entry of the i-th row) M[i,:] = M[i,:] - m*M[j,:] print("M[i,:] = ", M[i,:]) #Make sure that i-th row of M is correct (it's not!) return M  Testing with a 2x2 matrix A = np.array([[3,-2],[1,5]]) b = np.array([1,1]) gaussian_elim(A,b)  yields the following output: M[i,:] = [1 5 1] M[j,:] = [ 3 -2 1] m = 0.3333333333333333 M[i,:] - m*M[j,:] = [0. 5.66666667 0.66666667] <-- this is correct! M[i,:] = [0 5 0] <--why is this not equal to the above line??? array([[ 3, -2, 1], [ 0, 5, 0]])  The output I expected is array([[ 3, -2, 1],[0. 5.66666667 0.66666667]]). Why did the second row not update properly?

Answer: Because you use numpy array of integers, all your result will be rounded off to integers. You need to define A and b as A = np.array([[3,-2],[1,5]], dtype=np.float64) b = np.array([1,1], dtype=np.float64)  Doing so allows for float values in your matrices.
Question: So I am working on a small project and I can't for the life of me figure our why this doesn't work.... I am using a list for positional arguments, yet it returns that parametres are missing, I know its probably something basic but I can't seem to figure it out.. If is just place the write out the list direction in function it works, but it doesn't seem to want to work with the contesetants list. Hoping someone can help here! class Tester(): def __init__(self, first: int, second: int, third: int) -> None: self.first = first self.second = second self.third = third contestants = [54, 56, 32] print(Tester(contestants)) 

Answer: You need to unpack the list to pass the arguments: class Tester: def __init__(self, first: int, second: int, third: int) -> None: self.first = first self.second = second self.third = third def __str__(self) -> str: return f"Tester(first={self.first}, second={self.second}, third={self.third})" contestants = [54, 56, 32] print(Tester(*contestants))  Output: Tester(first=54, second=56, third=32)
Question: I have a react project that is hosted in flask app. This is the project folders: /simzol/ └── backend/ ├── app.py └── build/ ├── index.html ├── images/ ├── static/ │ ├── css/ │ └── js/ └── other_files...   PS. the build folder is generated by react using the command "yarn build"   here's the flask app initializer: app = Flask(__name__, static_url_path="/static", static_folder="build/static", template_folder="build")  that's how I serve the main route and any other path that isn't an API path (which should be handled by the react app. If the page doesn't exist, react should return the appropriate 404 not found page that I wrote): @app.route("/") @app.route('/<path:path>') def home(path=None): try: return render_template("index.html"), 200 except Exception as e: return handle_exception(e)  When I run the app and go to the main route, I can't see my images load up. What I tried:  move the images folder into the static folder. didn't change anything change the static_url_path to empty string ("") and the static_folder to "build". That solves the problem but I encounter another problem when I surf any page that is not the root page (like /contactus) directly through the browser's url input field, I get 404 error from flask (not react) I use relative image path in react src attributes, maybe changing that could fix the problem but if that solution works, I don't like it, because that makes the developing in react more complicated 

Answer: I wanted the images to load up in both flask app in production and in react app (using npm start) for development. At the same time, I wanted unknown pages to be handled by react. The most convenient solution I found is to put all the images under public/static/images instead of public/images and to change the src paths accordingly. Then I can run yarn build and get the result build folder to the flask app folder (backend). That worked perfectly.
Question: The package in question is the Dedalus spectral CFD library, here. For posterity I'll also link the project homepage. When I run conda search --channel conda-forge dedalus or variations with options I cannot find the dedalus package. This isn't a general issue with conda because I can find all default channels packages and every other conda-forge package I tested (most recently related libraries like fipy or gmsh, or general stuff like mamba). To be absolutely precise, this is what the terminal might look like on my end.  C:\Users\FirstName LastName>conda search --override-channels --channel conda-forge dedalus Loading channels: done No match found for: dedalus. Search: *dedalus* PackagesNotFoundError: The following packages are not available from current channels: - dedalus Current channels: - https://conda.anaconda.org/conda-forge/win-64 - https://conda.anaconda.org/conda-forge/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org and use the search bar at the top of the page.  I note that pip can locate the associated pypi project, but dedalus is very much a full-stack affair and so is best installed as a conda environment. I note also that my version of conda is 23.7.4. I note lastly that when I look for dedalus on an HPC server I SSH'd into I can find it, as one would hope. The server's version of conda is 4.8.3. I'm not sure why there's such a wide version difference, suggesting that maybe I'm on some deprecated system, but my listed build version is newer (3.26.1 vs 3.18.11). I would appreciate any assistance in troubleshooting this problem. I'm at a complete loss as to what my next steps should be.

Answer: Its not available on win64, check the OS tags, which only include  linux-64 osx-64 osx-arm64
Question: I want to iteratively compare 2 sets of rows in a PySpark dataframe, and find the common values in another column. For example, I have the dataframe (df) below. Column1 Column2 abc 111 def 666 def 111 tyu 777 abc 777 def 222 tyu 333 ewq 888  The output I want is abc,def,CommonRow <-- because of 111 abc,ewq,NoCommonRow abc,tyu,CommonRow <-- because of 777 def,ewq,NoCommonRow def,tyu,NoCommonRow ewq,tyu,NoCommonRow  The PySpark code that I'm currently using to do this is # "value_list" contains the unique list of values in Column 1 index = 0 for col1 in value_list: index += 1 df_col1 = df.filter(df.Column1 == col1) for col2 in value_list[index:]: df_col2 = df.filter(df.Column1 == col2) df_join = df_col1.join(df_col2, on=(df_col1.Column2 == df_col2.Column2), how="inner") if df_join.limit(1).count() == 0: # No common row print(col1,col2,"NoCommonRow") else: print(col1,col2,"CommonRow")  However, I found that this takes a very long time to run (df has millions of rows). Is there anyway to optimize it to run faster, or is there a better way to do the comparisons?

Answer: You can do this without loops using self join as follows: from pyspark.sql import SparkSession from pyspark.sql import functions as F spark = SparkSession.builder.getOrCreate() data = [("abc", 111), ("def", 666), ("def", 111), ("tyu", 777), ("abc", 777), ("def", 222), ("tyu", 333), ("ewq", 888)] df = spark.createDataFrame(data, ["Column1", "Column2"]) result_df = ( df.alias("a") .join(df.alias("b"), F.col("a.Column1") < F.col("b.Column1"), "left") # this is more efficient than the "!=" approach used in other answer because using "<" creates less rows after join (almost half) .join(df.alias("c"), (F.col("a.Column2") == F.col("c.Column2")) & (F.col("b.Column2") == F.col("c.Column2")), "left") .where(F.col("a.Column1").isNotNull() & F.col("b.Column1").isNotNull()) .groupBy("a.Column1", "b.Column1") .agg( F.when(F.count("c.Column2") > 0, "CommonRow").otherwise("NoCommonRow").alias("CommonStatus") ) .orderBy("a.Column1", "b.Column1") # remove this if not required ) result_df.show(truncate=False) # Output: # +-------+-------+------------+ # |Column1|Column1|CommonStatus| # +-------+-------+------------+ # |abc |def |CommonRow | # |abc |ewq |NoCommonRow | # |abc |tyu |CommonRow | # |def |ewq |NoCommonRow | # |def |tyu |NoCommonRow | # |ewq |tyu |NoCommonRow | # +-------+-------+------------+  Also, please avoid the orderBy when you run this on your million-row df - use it only if necessary.
Question: Given a 4D array of size (N,W,H,3), where N is the number of patches, W,H are the width and height of an image patch and 3 is the number of color channels. Assume that these patches were generated by taking and original image I and dividing it up into small squares. The order by which this division happen is row by row. So if we divide our image into 3x3 patches (9 total) each back is 10x10pixels, then the 4D array will be (9,10,10,3) and the order of element in it will be [patch11,patch12,patch13,patch21,patch22,patch23,patch31,patch32,patch33]. Now my question is about the most efficient way to combine these patches back to produce the original image in python only using simply python functions and numpy (no PIL or OpenCV). Thank you so much. I can write a double for loop that does the job as below, but I'm wondering if there is a better algorithm that can provide faster performance: import numpy as np def reconstruct_image(patches, num_rows, num_cols): # num_rows and num_cols are the number of patches in the rows and columns respectively patch_height, patch_width, channels = patches.shape[1], patches.shape[2], patches.shape[3] # Initialize the empty array for the full image full_image = np.zeros((num_rows * patch_height, num_cols * patch_width, channels), dtype=patches.dtype) # Iterate over the rows and columns of patches for i in range(num_rows): for j in range(num_cols): # Get the index of the current patch in the 4D array patch_index = i * num_cols + j # Place the patch in the appropriate position in the full image full_image[i*patch_height:(i+1)*patch_height, j*patch_width:(j+1)*patch_width, :] = patches[patch_index] return full_image N = 9 # Number of patches W, H, C = 10, 10, 3 # Patch dimensions (WxHxC) num_rows, num_cols = 3, 3 # Number of patches in rows and columns (3x3 patches) patches = np.random.rand(N, W, H, C) # Example patch data reconstructed_image = reconstruct_image(patches, num_rows, num_cols) 

Answer: Here's a fast pure numpy 1-liner way to do it: def reconstruct_image_2(): return patches.reshape(num_rows, num_cols, W, H, C).swapaxes(1, 2).reshape(num_rows*W, num_cols*H, C) reconstructed_image_2 = reconstruct_image_2() assert np.all(reconstructed_image == reconstructed_image_2) # True  Explanation: First reshape restructures your array as a "2D" array of patches, swapaxes makes your array (num_rows, W, num_cols, H, C), and finally the second and last reshape effectively concatenates the patches together in rows and columns. Timing comparison: import timeit %timeit reconstruct_image(patches, num_rows, num_cols) # 6.2 µs ± 16.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) %timeit reconstruct_image_2() # 1.56 µs ± 2.57 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
Question: For code that exists in a module named some.module and looks like this: class MyClass: @classmethod def method(cls): # do something pass  I'd like to know in the block marked as "do something" what the module name, the class name, and the method name are. In the above example, that would be:  module some.module class MyClass method method  I know of inspect.stack() and I can get the function / method name from it, but I can't find the rest of the data.

Answer: These are the most direct ways: import inspect class MyClass: @classmethod def method(cls): print("module:", __name__) print("class:", cls.__name__) print("method:", inspect.currentframe().f_code.co_name)  Doing the same from a utility function requires traversing "back" (up) in the call stack to find the calling frame: import inspect def dbg(): frame = inspect.currentframe().f_back print("module:", frame.f_globals.get("__name__")) print("cls:", frame.f_locals.get("cls")) print("method:", frame.f_code.co_name)
Question: I can't find a direct answer in NumPy documentation. This snippet will populate y with polynomial p values on domain x: p = [1, 2, 3] x = np.linspace(0, 1, 10) y = [np.polyval(p, i) for i in x]  What is the new API equivalent when p = Polynomial(p)?

Answer: You can simply evaluate values with p(x). Documentation can be found on "Using the convenience classes" under "Evaluation": p = [1, 2, 3] p = np.polynomial.Polynomial(p) x = np.linspace(0, 1, 10) y = p(x)  Note: Coefficients are in reverse order compared to legacy API i.e. coefficients go from lowest order to highest order such that p[i] is the ith-order term.
Question: I have a Django codebase that does a lot of Case/When/ExpressionWrapper/Coalesce/Cast ORM functions and some of them sometimes need a field as an argument - output_field. from django.db.models import FloatField, F some_param1=Sum(F('one_value')*F('second_value'), output_field=FloatField()) some_param2=Sum(F('one_value')*F('second_value'), output_field=FloatField()) some_param3=Sum(F('one_value')*F('second_value'), output_field=FloatField()) some_param4=Sum(F('one_value')*F('second_value'), output_field=FloatField()) some_param5=Sum(F('one_value')*F('second_value'), output_field=FloatField())  Sometimes I find myself wondering why am I always creating the same instance of any Field subclass over and over again. Is there any difference if I just pass one instance and share it between expressions? F.E from django.db.models import FloatField, F float_field = FloatField() some_param1=Sum(F('one_value')*F('second_value'), output_field=float_field) some_param2=Sum(F('one_value')*F('second_value'), output_field=float_field) some_param3=Sum(F('one_value')*F('second_value'), output_field=float_field) some_param4=Sum(F('one_value')*F('second_value'), output_field=float_field) some_param5=Sum(F('one_value')*F('second_value'), output_field=float_field)  I coulnd't find it in a documentation and the source code is not documented well regarding this parameter. P.S. The example is fake, just imagine a big annotate function that does a lot of processing using Case/When/ExpressionWrapper/Coalesce/Cast and has a lot of duplicated Field instances as output_field.

Answer: You can reuse the field. Using this output_field=… [Django-doc] serves two purposes:  the type sometimes requires specific formatting, typically for GIS columns, since a point, polygon, etc. needs to be converted to text so that Django can understand it; and to know what lookups transformations, etc. can be applied on it.  Indeed, if we use: queryset = queryset.annotate( some_param1=Sum( F('one_value') * F('second_value'), output_field=CharField() ) ) then Django will assume that some_param1 is a CharField (here this does not make much sense), and thus you can use: queryset.filter(some_param1__lower='a') since __lower is defined as a lookup on a CharField. But for a FloatField, it does not make much sense. But the field is not specialized or altered. It is thus more of a "signal" object to specify what can be done with it. That being said, I don't see much reasons to convert code to prevent constructing a FloatField. If we use %timeit, we get: In [1]: %timeit FloatField() 3.82 µs ± 379 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  So the construction takes approximately 3.82 microseconds. Typically a view has a lot more work to do than that, so writing a query that is itself more efficient, or saving a roundtrip to the database, will (very) likely outperform any optimization with respect to saving a FloatField by a few magnitudes.
Question: Given a DataFrame with categorical column: import polars as pl df = pl.DataFrame({ "id": ["a", "a", "a", "b", "b", "b", "b"], "value": [1,1,1,6,6,6,6], }) res = df.with_columns(bucket = pl.col.value.cut([1,3]))  shape: (7, 3) ┌─────┬───────┬───────────┐ │ id ┆ value ┆ bucket │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ cat │ ╞═════╪═══════╪═══════════╡ │ a ┆ 1 ┆ (-inf, 1] │ │ a ┆ 1 ┆ (-inf, 1] │ │ a ┆ 1 ┆ (-inf, 1] │ │ b ┆ 6 ┆ (3, inf] │ │ b ┆ 6 ┆ (3, inf] │ │ b ┆ 6 ┆ (3, inf] │ │ b ┆ 6 ┆ (3, inf] │ └─────┴───────┴───────────┘  How do I get all potential values of the categorical column? I can get them as strings with pl.Expr.cat.get_categories() as strings? res.select(pl.col.bucket.cat.get_categories())  shape: (3, 1) ┌───────────┐ │ bucket │ │ --- │ │ str │ ╞═══════════╡ │ (-inf, 1] │ │ (1, 3] │ │ (3, inf] │ └───────────┘  I can also get all existing values in their physical representation with pl.Expr.to_physical() res.select(pl.col.bucket.to_physical())  shape: (7, 1) ┌────────┐ │ bucket │ │ --- │ │ u32 │ ╞════════╡ │ 0 │ │ 0 │ │ 0 │ │ 2 │ │ 2 │ │ 2 │ │ 2 │ └────────┘  But how I can get all potential values in their physical representation? I'd expect output like: shape: (3, 1) ┌────────┐ │ bucket │ │ --- │ │ u32 │ ╞════════╡ │ 0 │ │ 1 │ │ 2 │ └────────┘  Or should I just assume that it's always encoded as range of integers without gaps?

Answer: I don't see any direct way. However, you could combine pl.Expr.cat.get_categories and pl.Expr.to_physical as follows. res.select( pl.col("bucket").cat.get_categories().cast(res.schema["bucket"]).to_physical() )  shape: (3, 1) ┌────────┐ │ bucket │ │ --- │ │ u32 │ ╞════════╡ │ 0 │ │ 1 │ │ 2 │ └────────┘  Here, it would be nice to have pl.Expr.meta.dtype implemented, such accessing res again can be avoided.