eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm attempting to add 100 documents to a Redis index, and then retrieving them with a match all query: import uuid import redis from redis.commands.json.path import Path import redis.commands.search.aggregation as aggregations import redis.commands.search.reducers as reducers from redis.commands.search.field import TextField, NumericField, TagField from redis.commands.search.indexDefinition import IndexDefinition, IndexType from redis.commands.search.query import NumericFilter, Query r = redis.Redis(host="localhost", port=6379) user4 = { "user": { "name": "Sarah Zamir", "email": "sarah.zamir@example.com", "age": 30, "city": "Paris", } } with r.pipeline(transaction=True) as pipe: for i, u in enumerate([user4] * 100): u["user"]["text"] = str(uuid.uuid4()) * 50 r.json().set(f"user:{i}", Path.root_path(), u) pipe.execute() schema = ( TextField("$.user.name", as_name="name"), TagField("$.user.city", as_name="city"), NumericField("$.user.age", as_name="age"), ) r.ft().create_index( schema, definition=IndexDefinition(prefix=["user:"], index_type=IndexType.JSON) ) result = r.ft().search(Query("*").paging(0, 100)) print(result.total) keys = r.keys("*") print(len(keys)) r.flushall() r.close()  I expect result.total to return 100, but it always returns inconsistent results less than 100. Am I doing something wrong? I checked the number of keys in redis and I get the correct count there.

Answer: Try to use pipe and add some time.sleep() import uuid import time import redis from redis.commands.json.path import Path from redis.commands.search.field import TextField, NumericField, TagField from redis.commands.search.indexDefinition import IndexDefinition, IndexType from redis.commands.search.query import NumericFilter, Query r = redis.Redis(host="localhost", port=6379) user4 = { "user": { "name": "Sarah Zamir", "email": "sarah.zamir@example.com", "age": 30, "city": "Paris", } } with r.pipeline(transaction=True) as pipe: for i, u in enumerate([user4] * 100): u["user"]["text"] = str(uuid.uuid4()) * 50 pipe.json().set(f"user:{i}", Path.root_path(), u) pipe.execute() schema = ( TextField("$.user.name", as_name="name"), TagField("$.user.city", as_name="city"), NumericField("$.user.age", as_name="age"), ) r.ft().create_index( schema, definition=IndexDefinition(prefix=["user:"], index_type=IndexType.JSON) ) while True: result = r.ft().search(Query("*")) if result.total == 100: break time.sleep(0.1) print(result.total) r.flushall() r.close()
Question: I have a scatter plot. I want to add some metadata to each point in my scatter plot. looking at documentation , i found annotate function( matplotlib.pyplot.annotate) , has anyone use this or some other function , such that we can add metadata to each point or are there are similar libraries, like matplotlib, which can display metadata , on clicking/hovering individual points in the scatterplot? import matplotlib.pyplot as plt import numpy as np # Generate random data for x and y axes x = np.random.rand(20) y = np.random.rand(20) colors = np.random.rand(20) area = (30 * np.random.rand(20))**2 # point radii # Create the scatter plot plt.scatter(x, y, s=area, c=colors, alpha=0.5) # Add labels and title plt.xlabel('X-axis') plt.ylabel('Y-axis') plt.title('Scatter Plot Example') # Display the plot plt.show() 

Answer: Matplotlib does not offer a built-in function in its core library to enable hover effects. For this functionality, you may consider using the mplcursor library. Kindly try running the code below after installing mplcursors. import matplotlib.pyplot as plt import numpy as np import mplcursors x = np.random.rand(20) y = np.random.rand(20) colors = np.random.rand(20) area = (30 * np.random.rand(20))**2 metadata = [f"Point {i}, Value: ({x[i]:.2f}, {y[i]:.2f})" for i in range(len(x))] fig, ax = plt.subplots() scatter = ax.scatter(x, y, s=area, c=colors, alpha=0.5) plt.xlabel('X-axis') plt.ylabel('Y-axis') plt.title('Interactive Scatter Plot') cursor = mplcursors.cursor(scatter, hover=True) @cursor.connect("add") def on_add(sel): sel.annotation.set_text(metadata[sel.index]) plt.show()  Output:
Question: I am breaking my head over this probably pretty simply question and I just can't find the answer anywhere. I want to create a new column with a grouped sum of another column, but I want to keep all individual rows. So, this is what the docs say: import polars as pl df = pl.DataFrame( { "a": ["a", "b", "a", "b", "c"], "b": [1, 2, 1, 3, 3], } ) df.group_by("a").agg(pl.col("b").sum())  The output of this would be: shape: (3, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═════╪═════╡ │ a ┆ 2 │ │ c ┆ 3 │ │ b ┆ 5 │ └─────┴─────┘  However, what I need would be this: shape: (5, 3) ┌─────┬─────┬────────┐ │ a ┆ b ┆ sum(b) │ │ --- ┆ --- ┆ ------ │ │ str ┆ i64 ┆ i64 │ ╞═════╪═════╪════════╡ │ a ┆ 1 ┆ 2 │ │ b ┆ 2 ┆ 5 │ │ a ┆ 1 ┆ 2 │ │ b ┆ 3 ┆ 5 │ │ c ┆ 3 ┆ 3 │ └─────┴─────┴────────┘  I could create the sum in a separate df and then join it with the original one, but I am pretty sure, there is an easier solution.

Answer: All you need is a window function: df.with_columns( b_sum=pl.col("b").sum().over(pl.col("a")) ) shape: (5, 3) ┌─────┬─────┬───────┐ │ a ┆ b ┆ b_sum │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 │ ╞═════╪═════╪═══════╡ │ a ┆ 1 ┆ 2 │ │ b ┆ 2 ┆ 5 │ │ a ┆ 1 ┆ 2 │ │ b ┆ 3 ┆ 5 │ │ c ┆ 3 ┆ 3 │ └─────┴─────┴───────┘
Question: I inherited a 1000+ line query the works in oracle's sql developer. When I try running the query in a python script, I get an error (An error occurred: ORA-00900: invalid SQL statement) at the line: EXEC DBMS_STATS.gather_table_stats('SSS', 'YYY'). I have very little experience with databases and a Google search of the error has not yielded a possible solution. Python script:  with open('myquery.sql', 'r') as sql_file: query = str(sql_file.read().replace("\n\n","")) sql_commands = query.split(';') for command in sql_commands: try: if command.strip() != '': print(command) cur.execute(command) connection.commit() except Exception as e: print(f"An error occurred: {e}") connection.commit()  The last few lines of the query(myquery.sql): DELETE FROM CD_SS.MM_DSD_MIG_MASTER; commit; INSERT INTO CD_SS.MM_DSD_MIG_MASTER SELECT DISTINCT PATTERN, b_ITEM_NUM, 'N', NULL, NULL, NULL FROM CD_SS.MM_DSD_MIG_PATTERNS; commit; update MM_DSD_MIG_MASTER set MIGRATION_STATUS_FLAG = 'Z' where MIGRATION_STATUS_FLAG = 'N'; commit; grant all privileges on MM_DSD_MIG_MASTER to public; grant all privileges on MM_DSD_MIG_PATTERNS to public; DBMS_STATS.gather_table_stats('CD_SS', 'MM_DSD_MIG_PATTERNS'); DBMS_STATS.gather_table_stats('CD_SS', 'MM_DSD_MIG_MASTER');  Everything works if the DMS_STATS lines are omited but was told it has to stay.

Answer: The script can be converted into a single PL/SQL anonymous block. The advantage of this conversion is that it allows the exact same block to run on all environments. (Although the slash at the end should probably be excluded when running from Python.) The disadvantage of this conversion is that all the DDL statements need to be converted into EXECUTE IMMEDIATE statements. BEGIN DELETE FROM CD_SS.MM_DSD_MIG_MASTER; commit; INSERT INTO CD_SS.MM_DSD_MIG_MASTER SELECT DISTINCT PATTERN, b_ITEM_NUM, 'N', NULL, NULL, NULL FROM CD_SS.MM_DSD_MIG_PATTERNS; commit; update MM_DSD_MIG_MASTER set MIGRATION_STATUS_FLAG = 'Z' where MIGRATION_STATUS_FLAG = 'N'; commit; EXECUTE IMMEDIATE 'grant all privileges on MM_DSD_MIG_MASTER to public'; EXECUTE IMMEDIATE 'grant all privileges on MM_DSD_MIG_PATTERNS to public'; DBMS_STATS.gather_table_stats('CD_SS', 'MM_DSD_MIG_PATTERNS'); DBMS_STATS.gather_table_stats('CD_SS', 'MM_DSD_MIG_MASTER'); END; /
Question: Is it possible to do something like this in Polars? Like do you need a separate when.then.otherwise for each of the 4 new varialbles, or can you use struct to create multiple new variables from one when.then.otherwise? Regular Python example: if x=1 and y=3 and w=300*z and z<100: tot = 300 work = 400 sie = 500 walk = 'into' else: tot = 350 work = 400*tot sie = tot/1000 walk = 'outof'  I tried to do a similar thing in Polars with struct (to create new variables a and b based on Movie variable: import polars as pl ratings = pl.DataFrame( { "Movie": ["Cars", "IT", "ET", "Cars", "Up", "IT", "Cars", "ET", "Up", "Cars"], "Theatre": ["NE", "ME", "IL", "ND", "NE", "SD", "NE", "IL", "IL", "NE"], "Avg_Rating": [4.5, 4.4, 4.6, 4.3, 4.8, 4.7, 4.5, 4.9, 4.7, 4.6], "Count": [30, 27, 26, 29, 31, 28, 28, 26, 33, 28], } ) x = ratings.with_columns( pl.when(pl.col('Movie')=='Up').then(pl.struct(pl.lit(0),pl.lit(2))).otherwise(pl.struct(pl.lit(1),pl.lit(3))).struct.field(['a','b']) ) print(x)  Thanks!

Answer: If you remove the .struct.field() call you will see the issue. # DuplicateError: multiple fields with name 'literal' found  You need to give names to the struct fields you are creating. df.with_columns( pl.when(pl.col('Movie') == 'Up') .then(pl.struct(pl.lit(0).alias('a'), pl.lit(2).alias('b'))) .otherwise(pl.struct(pl.lit(1).alias('a'), pl.lit(3).alias('b'))) .struct.field('a', 'b') )  There are also some ways to neaten it up if you prefer.  pl.when() uses kwargs as shorthand for equality conditions. pl.struct() uses kwargs as shorthand for .alias() to name fields. pl.lit() is not required for integers .struct.unnest() will unnest all fields.  df.with_columns( pl.when(Movie='Up') .then(pl.struct(a=0, b=2)) .otherwise(pl.struct(a=1, b=3)) .struct.unnest() )  shape: (10, 6) ┌───────┬─────────┬────────────┬───────┬─────┬─────┐ │ Movie ┆ Theatre ┆ Avg_Rating ┆ Count ┆ a ┆ b │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ i64 ┆ i32 ┆ i32 │ ╞═══════╪═════════╪════════════╪═══════╪═════╪═════╡ │ Cars ┆ NE ┆ 4.5 ┆ 30 ┆ 1 ┆ 3 │ │ IT ┆ ME ┆ 4.4 ┆ 27 ┆ 1 ┆ 3 │ │ ET ┆ IL ┆ 4.6 ┆ 26 ┆ 1 ┆ 3 │ │ Cars ┆ ND ┆ 4.3 ┆ 29 ┆ 1 ┆ 3 │ │ Up ┆ NE ┆ 4.8 ┆ 31 ┆ 0 ┆ 2 │ │ IT ┆ SD ┆ 4.7 ┆ 28 ┆ 1 ┆ 3 │ │ Cars ┆ NE ┆ 4.5 ┆ 28 ┆ 1 ┆ 3 │ │ ET ┆ IL ┆ 4.9 ┆ 26 ┆ 1 ┆ 3 │ │ Up ┆ IL ┆ 4.7 ┆ 33 ┆ 0 ┆ 2 │ │ Cars ┆ NE ┆ 4.6 ┆ 28 ┆ 1 ┆ 3 │ └───────┴─────────┴────────────┴───────┴─────┴─────┘
Question: Essentially, I want to group all records that match either key Input: |id|key1|key2| |-|-|-| |1|a|x| |2|a|y| |3|b|y| |4|c|z|  Desired output: (The first 3 rows are grouped together) |key1 (or any identifier, I only care about the final counts)|len| |-|-| |a|3| |c|1|  My attempted (incomplete) solution: import polars as pl df = pl.DataFrame({ 'id': [1, 2, 3, 4], 'key1': ['a', 'a', 'b', 'c'], 'key2': ['x', 'y', 'y', 'z'], }) df = ( df.group_by( 'key1', 'key2', ) .len() .group_by('key1') .agg( pl.col('key2'), pl.col('len').sum() ) ) print(df)  Which gives: |key1|key2|len| |-|-|-| |a|["y", "x"]|2| |b|["y"]|1| |c|["z"]|1|  However, I'm not sure how I would further group this by key2 (merging the a and b rows, since they have a common value of y) while preserving the sum of len

Answer: If I understand you correctly, you want to create equivalence relationships between key1 and key2, and keep merging things into groups if there exists a relationship chain that connects the two? So in this case since there is a row a, y any rows which have a or y in them should be merged? In this case it's hard to express this directly in Polars, but you can do a hybrid with scipy's DisjointSet. First, gather all the unique keys and relationships: keys = df.select(pl.col.key1.append(pl.col.key2).unique()) edges = df.select(["key1", "key2"]).unique()  Then, insert them into a DisjointSet and merge into groups: from scipy.cluster.hierarchy import DisjointSet ds = DisjointSet(keys.to_series().to_list()) for a, b in edges.iter_rows(): ds.merge(a, b)  Then convert the DisjointSet into a DataFrame mapping key to group index: group_idx = ( pl.DataFrame({"key": [list(s) for s in ds.subsets()]}) .lazy() .with_row_index("group_idx") .explode("key") )  Finally, we join the group_idx to our original dataframe and group by the group index: (df.lazy() .join(group_idx, left_on="key1", right_on="key") .group_by("group_idx") .len() .collect())  Thus giving our output: ┌───────────┬─────┐ │ group_idx ┆ len │ │ --- ┆ --- │ │ u32 ┆ u32 │ ╞═══════════╪═════╡ │ 0 ┆ 1 │ │ 1 ┆ 3 │ └───────────┴─────┘
Question: I am training a random forest classifier in python sklearn, see code below- from sklearn.ensemble import RandomForestClassifier rf = RandomForestClassifier(random_state=42) rf.fit(X = df.drop("AP", axis =1), y = df["AP"].astype(int))  When I predict the values using this classifier on another dataset that has NaN values, the model provides some output. Not even that, I tried predicting output on a row with all variables as NaNs, it predicted the outputs. #making a row with all NaN values row = pd.DataFrame([np.nan] * len(rf.feature_names_in_), index=rf_corn.feature_names_in_).T rf.predict(row)  It predicts- array([1]) I know that RandomForestClassifier in scikit-learn does not natively support missing values. So I expected a ValueError, not a prediction. I can ignore the NaN rows and only predict on non-nan rows but I am concerned if there is something wrong with this classifier. Any insight will be appreciated.

Answer: In the most recent version of scikit-learn (v1.4) they added support for missing values to RandomForestClassifier when the criterion is gini (default). Source: https://scikit-learn.org/dev/whats_new/v1.4.html#id7
Question: I am trying to create a token with FastAPI: > import json > import os > import aiohttp > import asyncio > from fastapi import FastAPI > from fastapi import APIRouter, Request > from fastapi.responses import JSONResponse > > token = APIRouter(prefix="/management/api", tags=["API Apl token"]) > > app=FastAPI() > app.include_router(token)  and many methods after... I got this > uvicorn peg:token --reload > INFO: Will watch for changes in these directories: ['C:\\Users\\Ejbc25\\fastapi'] > INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) > INFO: Started reloader process [2232] using StatReload > INFO: Started server process [20632] > INFO: Waiting for application startup. > INFO: Application startup complete. > INFO: 127.0.0.1:51183 - "GET / HTTP/1.1" 404 Not Found > INFO: 127.0.0.1:51183 - "GET / HTTP/1.1" 404 Not Found  How to fix this problem? I expected that http://127.0.0.1:8000/token would return token but shows  Not Found 

Answer: The reason of your issue is that you're trying to run a FastAPI APIRouter instance (token) directly with uvicorn, instead of the actual FastAPI application (app). Your uvicorn command: uvicorn peg:token --reload  is trying to start the app using the token router as the main app, which won’t work since token is not a FastAPI application (it’s just a router meant to be included in an app). You should point uvicorn to the actual FastAPI instance, which in your code is named app. Change your command to: uvicorn peg:app --reload  Obviously I'm assuming that the your file is named peg.py, but if it has a different name update the command (e.g., uvicorn main:app --reload for main.py).
Question: In production deployment of Hugging Face LLMs, the first inference call often has very high latency ("cold start"), even on a machine where the model is already loaded into memory. Subsequent calls are much faster. I want to implement a model warm-up strategy that:  Primes the model and GPU memory before real user requests arrive Reduces first-token generation time for users Works for both pipeline()-based and model.generate()-based inference  from transformers import pipeline generator = pipeline('text-generation', model="tiiuae/falcon-7b-instruct", device=0) def generate_text(prompt): return generator(prompt, max_new_tokens=50)[0]['generated_text']  My Question: What is the best way to warm up a HuggingFace Transformers model after loading, to minimize first-token latency in production inference?

Answer: You could use a dummy inference immediately after loading the model. For pipeline: # Warmup _ = generator("Warm up prompt", max_new_tokens=1)  For raw model.generate(): from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b-instruct") model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-7b-instruct", device_map="auto", torch_dtype="auto") # Warmup inputs = tokenizer("Warm up prompt", return_tensors="pt").to(model.device) _ = model.generate(**inputs, max_new_tokens=1)
Question: In a Python script, I've written: # etc. etc. input_file = args.input_file_path or sys.stdin arr = numpy.fromfile(input_file, dtype=numpy.dtype('f32'))  when I run the script, I get: $ cat nums.fp32.bin | ./myscript File "./myscript", line 123, in main arr = numpy.fromfile(input_file, dtype=numpy.dtype('f32')) OSError: obtaining file position failed  why does NumPy need the file position? And - can I circumvent this somehow?

Answer: This error happens because np.fromfile() is implemented in a fairly counterintuitive way. You might assume that this is implemented by repeatedly calling e.g. file.read(4096), then copying the resulting buffer to the appropriate place in the array. It does not work like this. Instead, it is following roughly this process:  Find the file descriptor number of the file object. Copy that file descriptor using os.dup() in Python. Find the read position within the original file by calling f.tell() in Python. Set the copied file descriptor to the same read position using fseek() in C.  At the end of this process, NumPy has a C-level file that it owns, and can copy data from without the overhead of calling a Python method. It then reads the file and copies it into the array. (You may be asking why steps 3 and 4 are necessary. Doesn't copying a file descriptor copy its read position? This is true, but it won't work if the Python file is buffered, as the C-level read position and Python-level read position may not match.) To clean up this file descriptor, NumPy does the following.  Find the seek position of the C-level file. Copy the seek position to the Python-level file. Close the C-level file.  In order for np.fromfile() to work, your file-like object must support all of the following:  It must have a file descriptor. This rules out, for example, BytesIO, because that is not backed by an OS-level file. It must support seek(). This rules out the use of pipes. It must support tell(). It must support flush().  In practice, this rules out most file-like objects that are not really files. To learn more about this, I recommend reading the source code.  And - can I circumvent this somehow?  No. All four of those things are mandatory. You can work around it, however. Assuming the file f is open in binary mode, you could do f.read() and obtain a bytes object. You can then pass this to object to np.frombuffer() to obtain an array.
Question: I have this function which converts an English month to a French month: def changeMonth(month): global CurrentMonth match month: case "Jan": return "Janvier" case "Feb": return "Février" case "Mar": return "Mars" case "Apr": return "Avril" case "May": return "Mai" case "Jun": return "Juin" case "Jul": return "Juillet" case "Aug": return "Août" case "Sep": return "Septembre" case "Oct": return "Octobre" case "Nov": return "Novembre" case "Dec": return "Décembre" # If an exact match is not confirmed, this last case will be used if provided case _: return ""  and I have a pandas col df["month"]= df['ic_graph']['month'].tolist():  now what I'm looking for is to pass the df["month"] col through the changeMonth function to display the df["month"] in frensh months By the way, I do not want to use the >>> import locale >>> locale.setlocale(locale.LC_ALL, 'fr_FR') 

Answer: Question seems a bit unclear to me, Do you want to replace the month column in dataframe itself? If yes something like this can work: def changeMonth(month): if month=="Jan": return "Janvier" elif month=="Feb": return "Février" elif month=="Mar": return "Mars" elif month=="Apr": return "Avril" elif month=="May": return "Mai" elif month=="Jun": return "Juin" elif month=="Jul": return "Juillet" elif month=="Aug": return "Août" elif month=="Sep": return "Septembre" elif month=="Oct": return "Octobre" elif month=="Nov": return "Novembre" elif "Dec": return "Décembre" # If an exact match is not confirmed, this last case will be used if provided else: return "" df['month']=df['month'].apply(changeMonth)  But if you want to change the month while just printing the list out, you can use list comprehension, french_month_list = [changeMonth(i) for i in df['ic_graph']['month'].tolist()]
Question: In my simple Flask app, I've created the following .flaskenv file: FLASK_APP=simpleflask.app FLASK_RUN_HOST=127.0.0.1 FLASK_RUN_PORT=60210  Now, I would like my app to log the following message upon startup: Running my.flask.app on http://127.0.0.1:60210 ...  The message should be logged just once, not per each request. Preferably, I would like to avoid parsing .flaskenv on my own, but rather use some internal Flask objects and obtain this information dynamically. Any idea how to achieve this?

Answer: app = Flask(__name__) # ... with app.app_context(): app.logger.info(f"Running {os.environ.get('FLASK_APP')} on http://{os.environ.get('FLASK_RUN_HOST')}:{os.environ.get('FLASK_RUN_PORT')} ...")
Question: I'm trying to create a Python function that generates a valid key in my Firebase Realtime Database (using the same logic as other keys, ensuring no collision and chronological order). Something like this: def get_new_key() -> str: return db.reference(app=firebase_app).push().key print(get_new_key()) # output: '-OOvws9uQDq9Ozacldjr'  However, doing this actually creates the node with an empty string value, as specified in the doc. Is there any way to grab the key without adding anything to the database? My only thought right now would be to reimplement myself their key generation algorithm, but this looks overkill and not future-proof if this algorithm changes.

Answer: You're right, each time you call push(value='') a child node in the Realtime Database will be created. If you are calling the function without passing a value, the default string that will be used for writing the data will be an empty string. As far as I know, the Firebase SDKs for Python does not publicly expose a push() function that only generates the IDs. So the only option that you have would be to use the algorithm that is present below:  https://gist.github.com/mikelehen/3596a30bd69384624c11  It is written in JavaScript, but I think that you can adapt it very easily to Python. This algorithm is very stable, and I don't think that it will change because many clients (Android, iOS, Web) depend on it. Here's a the Python implementation of Firebase’s generatePushID() (not tested): import time import random PUSH_CHARS = '-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz' def generate_push_id(): now = int(time.time() * 1000) time_stamp_chars = [] for _ in range(8): time_stamp_chars.append(PUSH_CHARS[now % 64]) now //= 64 time_stamp_chars.reverse() push_id = ''.join(time_stamp_chars) for _ in range(12): push_id += random.choice(PUSH_CHARS) return push_id