eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've used Polars for some time now but this is something that often makes me go from Polars DataFrames to native Python calculations. I've spent resonable time looking for solutions that (tries) to use shift(), rolling(), group_by_dynamic() and so on but none is successful. Task Do calculation that depends on previous calculation's result that is in the same column. Example in Excel In Excel this is like the most straighforward formula ever...if the "index" is zero I want to return "A", otherwise I want to return the result from the cell above.     A B C     1 Index Result Formula for the "Result" column   2 0 A =IF(A2=0;"A";B1)   3 1 A =IF(A3=0;"A";B2)    Where is the recursion In column "B" the formula refers to the previously calculated values on the same column "B".  Copy & Paste Excel's solution to Polars # Import Polars module. import polars as pl # Create the data. data = {'Index': [0, 1]} # Create the DataFrame. df = pl.from_dict(data) # Add a column to the DataFrame. df = df.with_columns( # Tries to reproduce the Excel formula. Result = pl.when( pl.col('Index') == 0 ).then( pl.lit('A') ).otherwise( pl.col('Result') ) )  The issue Within the "with_columns()" method the "Result" column cannot be referred because It doens't exist in the DataFrame yet. If we try to do so, we get a ColumnNotFoundError:  Question Any idea on how can I accomplish such a simple task on Polars? Thank you,

Answer:  How can I do a calculation that depends on previous (row's) calculation result that is in the same column?  The short answer is that you can't without falling back into Python. To do this, any library would need to essentially need to iterate over the rows, only calculating a single row at a time. This means any sort of vectorisation is not possible. Polars offers map_elements for this use-case, but it is discouraged. From the docs:  This method is much slower than the native expressions API. Only use it if you cannot implement your logic otherwise.  df = pl.DataFrame({'Index': [1, 0, 1, 1, 0]}) previous_result = "Result" # hardcode the header as the intial "previous result" def f(index): global previous_result out = "A" if index == 0 else previous_result previous_result = out return out print(df.with_columns(Result=pl.col("Index").map_elements(f, return_dtype=pl.String))) # shape: (5, 2) # ┌───────┬────────┐ # │ Index ┆ Result │ # │ --- ┆ --- │ # │ i64 ┆ str │ # ╞═══════╪════════╡ # │ 1 ┆ Result │ # │ 0 ┆ A │ # │ 1 ┆ A │ # │ 1 ┆ A │ # │ 0 ┆ A │ # └───────┴────────┘  The better solution is to attempt to recognise any pattern that allows the computation to be done in a vectorised way. In this (likely contrived) example, it is that once Index == 0 has been seen once, the result of the remainder of the column is "A" df.with_columns( # If the row number is >= to the first row an index of 0 was seen pl.when(pl.int_range(pl.len()) >= pl.arg_where(pl.col("Index") == 0).min()) .then(pl.lit("A")) .otherwise(pl.lit("Result")) .alias("Result") ) # Returns the same output as above
Question: According to GitHub syntax highlighting, keyword for pip requirements syntax highlighting can be found on languages.yml. According to the link, the keyword is Pip Requirements, but the following markdown snippet isn't highlighted on GitHub: ```Pip Requirements pandas==2.2.3 ```  How to syntax highlight pip requirements?

Answer: Use "pip-requirements" instead of "Pip Requirements":  ```pip-requirements --editable . foo==1.2 bar==3.4 baz[quux]>=1.0.1 ```  It does not seem to render on Stack Overflow, but here is a screencap of markdown rendered from Github:  Source: https://gist.github.com/wimglenn/204dd744240848d8cbf2b9beb6eb4a83
Question: Given the below code I am expecting the keyboard to press the page down key. However, instead it is pressing the number 3. All other keys I have used appear to be working correctly. How do I get this mapped correctly? import time import pydirectinput time.sleep(2) pydirectinput.keyDown('pagedown') time.sleep(0.01) pydirectinput.keyUp('pagedown') 

Answer: You want the sequence of keystrokes to include NumLock.
Question: I'm using Python protobuf package version 5.38.3 for deserializing some packets and I need to check if the messages I deserialize are conformant or not to a specific protobuf message structure. For some checks I want to obtain the list of unknown fields. This post points to an API UnknownFields() supported by messages, but when I call it in a deserialized message it raises NotImplementedError. How can I get access to the list of unknown fields from a deserialized message in protobuf 5.28.3?

Answer:  How can I get access to the list of unknown fields  Here, let me google that for you. https://protobuf.dev/news/2023-08-15  Python Breaking Change In v25 message.UnknownFields() will be deprecated in pure Python and C++ extensions. It will be removed in v26. Use the new UnknownFieldSet(message) support in unknown_fields.py as a replacement.  You will want to update your code to use the new public API.
Question: We have data from a REST API call stored in an output file that looks as follows: Sample Input File: test test123 - test (bla bla1 (On chutti)) test test123 bla12 teeee (Rinku Singh) balle balle (testagain) (Rohit Sharma) test test123 test1111 test45345 (Surya) (Virat kohli (Lagaan)) testagain blae kaun hai ye banda (Ranbir kapoor (Lagaan), Milkha Singh (On chutti) (Lagaan))  Expected Output: bla bla1 Rinku Singh Rohit Sharma Virat kohli Ranbir kapoor, Milkha Singh  Conditions to Derive the Expected Output:  Always consider the last occurrence of parentheses () in each line. We need to extract the values within this last, outermost pair of parentheses. Inside the last occurrence of (), extract all values that appear before each occurrence of nested parentheses (). Eg: test test123 - test (bla bla1 (On chutti)) last parenthesis starts from (bla to till chutti)) so I need bla bla1 since its before inner (On chutti). So look for the last parenthesis and then inside how many pair of parenthesis comes we need to get data before them, eg: in line testagain blae kaun hai ye banda (Ranbir kapoor (Lagaan), Milkha Singh (On chutti) (Lagaan)) needed is Ranbir kapoor and Milkha Singh.  Attempted Regex: I tried using the following regular expression on Working Demo of regex: Regex: ^(?:^[^(]+\([^)]+\) \(([^(]+)\([^)]+\)\))|[^(]+\(([^(]+)\([^)]+\),\s([^\(]+)\([^)]+\)\s\([^\)]+\)\)|(?:(?:.*?)\((.*?)\(.*?\)\))|(?:[^(]+\(([^)]+)\))$  The Regex that I have tried is working fine but I want to improve it with the advice of experts here. Preferred Languages: Looking to improve this regex OR a Python, or an awk answer is also ok. I myself will also try to add an awk answer.

Answer: Purely based on your shown input and your comments reflecting that you need to capture 1 or 2 values per line, here is an optimized regex solution: ^(?:\([^)(]*\)|[^()])*\(([^)(]+)(?:\([^)(]*\)[, ]*(?:([^)(]+))?)?  RegEx Demo RegEx Details: This regex solution does the following:  match everythng before last (...) then match ( then 1st group: match name that must not have ( and ) then optional match of (...) or comma/space then 2nd group: match name that must not have ( and )  Further Details:  ^: Start (?:: Start non-capture group  \([^\n)(]*\): Match any pair of (...) text |: OR [^()\n]: Match any character that are not (, ) and \n   )*: End non-capture group. Repeat this 0 or more times \(: Match last ( ([^)(\n]+): 1st capture group that matches text with 1+ characters that are not (, ) and \n (?:: Start non-capture group 1  \([^\n)(]*\): Match any pair of (...) text [, ]*: Match 0 or more of space or comma characters (?:: Start non-capture group 2  ([^)(\n]+): 2nd capture group that matches text with 1+ characters that are not (, ) and \n   )?: End non-capture group 2. ? makes this an optional match   )?: End non-capture group 1. ? makes this an optional match
Question: After a much-appreciated assist from @zyd in this answer to parallel, deep-first execution in Dagster, I am now looking for a way to run an @op on the collected results of the graph run, or at least one that waits until they have all finished, since they don't have hard dependencies per se. My working code is as follows: @op def get_csv_filenames(context) -> List[str]: @op(out=DynamicOut()) def generate_subtasks(context, csv_list:List[str]): for csv_filename in csv_list: yield DynamicOutput(csv_filename, mapping_key=csv_filename) @op # no dep since 1st task def load_csv_into_duckdb(context, csv_filename) @op(ins={"start":In(Nothing)} def transform_dates(context, csv_filename) @op(ins={"start":In(Nothing)} def from_code_2_categories(context, csv_filename) @op(ins={"start":In(Nothing)} def export_2_parquet(context, csv_filename) @op(ins={"start":In(Nothing)} def profile_dataset(context, csv_filename) @graph def process(context, csv_filename:str): task1 = load_csv_into_duckdb(context=context, csv_filename=csv_filename) task2 = transform_dates(start=task1, context=context, csv_filename=csv_filename) task3 = from_code_2_categories(start=task2, context=context, csv_filename=csv_filename) task4 = export_2_parquet(start=task3, context=context, csv_filename=csv_filename) profile_dataset(start=task4, context=context, csv_filename=csv_filename) @job def pipeline(): csv_filename_list = get_csv_filenames() generate_subtasks(csv_filename_list).map(process)  I have tried the .map(process).collect() approach, but Dagster complains that Nonetype has no attribute collect. However, I've seen several examples online of this same approach and apparently it should work. I have also tried for the @graph to return a list of the individual task return values, but DagsterUI complains that a graph-decorated function should return a dict with mapping keys. I could build that, but I feel I should instead pick that up from Dagster's execution context, which I don't know how to access from within the graph funct. Does anyone have some pointers?

Answer: Here's an example that worked for me: from dagster import Definitions, op, DynamicOutput, graph, GraphOut, DynamicOut @op def a_op(path): return path @op def op2(path): return path @op def op3(path): return path @op(out=DynamicOut(str)) def mapper(): for i in range(10): yield DynamicOutput(str(i), mapping_key=str(i)) # I think what you were missing is returning the output from the graph here @graph(out={"out": GraphOut()}) def nested(path: str): return op2(a_op(path)) @op def consumer(context, paths: list[str]): context.log.info(paths) @graph def the_graph(): consumer(mapper().map(nested).collect()) the_job = the_graph.to_job() defs = Definitions( jobs=[the_job], )  A graph is really just an organizational concept for grouping ops. At runtime nested graphs are flattened into a single graph for the whole job. This implies that if you want to use the output of an op in the nested graph, you need to return the output of the op from the nested graph. Then everything else works the same as any other op->op dependency.
Question: I'm trying to add a String type data into a postgresql database in Python by Prisma. The culumn in database is defined as a specific enum type in Prisma schema. I tried to insert by a String mapping. However the insert was failed with not support with String type. What should I do? from enum import Enum from prisma import Prisma db = Prisma() os.environ['DATABASE_URL'] = '' db.connect() class DifficultyLevel(Enum): EASY = 'EASY' MEDIUM = 'MEDIUM' HARD = 'HARD' LUCK = 'LUCK' def map_question_difficulty(generated_difficulty): difficulty_mapping = { 'Easy': DifficultyLevel.EASY, 'Medium': DifficultyLevel.MEDIUM, 'Hard': DifficultyLevel.HARD, 'LUCK': DifficultyLevel.LUCK } return difficulty_mapping.get(generated_difficulty, DifficultyLevel.EASY).value # Default to EASY if difficulty not found async def insert_questions_to_db(questions): try: for question in questions: db.elementgenerated.create( data={ "difficulty": map_question_difficulty(question['difficulty']) } ) logging.info(f"Successfully inserted {len(questions)} questions into the database.") except Exception as e: logging.error(f"Error inserting questions into the database: {e}") raise  Definition in Prisma schema: enum DifficultyLevel { EASY MEDIUM HARD }  Here the Program gave feedback with: 'prisma.errors.DataError: Error converting field "difficulty" of expected non-nullable type "String", found incompatible value of "EASY".' The question['difficulty'] is a String object like 'Easy', 'Medium' or 'Hard'.

Answer: missing enumerated value Your prisma schema should also include LUCK. A discrepancy between python and prisma seems like a Bad Thing. read the diagnostic  prisma.errors.DataError: Error converting field "difficulty" of expected non-nullable type "String", found incompatible value of "EASY".' The question['difficulty'] is a String object like 'Easy', 'Medium' or 'Hard'.  This suggests revising your Enum class to be: class DifficultyLevel(Enum): EASY = 'Easy' MEDIUM = 'Medium' HARD = 'Hard' LUCK = 'Luck'  auto() Side note: consider DRYing up the class by defining it this way: from enum import Enum, auto class DifficultyLevel(Enum): EASY = auto() MEDIUM = auto() HARD = auto() LUCK = auto() def __str__(self): return self.name.title()  And then use f-string or str() rather than .value.
Question: To be more specific, I'm wondering how to assign a tensor by slice and by mask at different dimension(s) simultaneously in PyTorch. Here's a small example about what I want to do: With the tensors and masks below: x = torch.zeros(2, 3, 4, 6) mask = torch.tensor([[ True, True, False], [True, False, True]]) y = torch.rand(2, 3, 1, 3)  I want to achieve something like x[mask, :, :3] = y[mask]   In dimension 0 and 1, only the 4x6/1x3 slices in x/y that whose corresponding element in mask is True are allowed to be assigned. In dimension 2, I hope the 1-row tensor in y can be broadcast to all the 8 rows of x, In dimension 3, only the first 3 elements in x are assigned with the 3-element tensor from y.  However, with code above, following error was caught: RuntimeError: shape mismatch: value tensor of shape [4, 1, 3] cannot be broadcast to indexing result of shape [4, 3, 6]  It seems that PyTorch did [mask] indexing instead, and ignored the :3 indexing. I've also tried x[mask][:, :, :3] = y[mask]  No error occurred but the assignment still failed. I know I can assign by slice and by mask step by step, but I hope to avoid any intermediate tensors if possible. Tensors in neural networks may be extremely big, so may be an all-in-one assignment may take less time and less memory.

Answer: You can do the following: x = torch.zeros(2, 3, 4, 6) mask = torch.tensor([[ True, True, False], [True, False, True]]) y = torch.rand(2, 3, 1, 3) x[..., :3][mask] = y[mask]  This produces the same result as i, j = mask.nonzero(as_tuple = True) x[i, j, :, :3] = y[i, j]  For the 2D mask scenario. This method also works for additional dims: x = torch.zeros(2, 3, 3, 4, 6) y = torch.rand(2, 3, 3, 1, 3) mask = torch.rand(2,3,3)>0.5 x[..., :3][mask] = y[mask]  For additional dims, the only constraint is that the first n=mask.ndim dims of x and y must match the shape of mask and the final dimension of y is 3 to match the :3.
Question: I am new to using JAX, and I’m still getting familiar with how it works. From what I understand, when using Just-In-Time (JIT) compilation (jax.jit), the first execution of a function might be slower due to the compilation overhead, but subsequent executions should be faster. However, I am seeing the opposite behavior. In the following code snippet: from icecream import ic import jax from time import time import numpy as np @jax.jit def my_function(x, y): return x @ y vectorized_function = jax.vmap(my_function, in_axes=(0, None)) shape = (1_000_000, 1_000) x = np.ones(shape) y = np.ones(shape[1]) start = time() vectorized_function(x, y) t_1 = time() - start start = time() vectorized_function(x, y) t_2 = time() - start print(f'{t_1 = }\n{t_2 = }')  I get the following results: t_1 = 13.106784582138062 t_2 = 15.664098024368286  As you can see, the second run (t_2) is actually slower than the first one (t_1), which seems counterintuitive to me. I expected the second run to be faster due to JAX’s JIT caching. Has anyone encountered a similar situation or have any insights into why this might be happening? PS: I know I could have done x @ y directly without invoking vmap, but this is an easy example just to test its behaviour. My actual code is more complex, and the difference in runtime is even bigger (around 8x slower). I hope this simple example works similar.

Answer: For general tips on running JAX microbenchmarks effectively, see FAQ: Benchmarking JAX code. I cannot reproduce the timings from your snippet, but in your more complicated case, I suspect you are getting fooled by JAX's Asynchronous dispatch, which means that the timing method you're using will not actually reflect the time taken by the underlying computation. To address this, you can wrap your results in jax.block_until_ready: start = time() vectorized_function(x, y).block_until_ready() t_1 = time() - start
Question: I've been working through Python Crash Course e2, and got stuck on Ch.18. Having opened models.py, and entered the code, the error message is: ModuleNotFoundError: No module named 'django' I have spent some time working on this, without a solution. Could it be that PCCe2 is out of date, or is there a workaround solution? ChatGPT has said to make sure to activate the virtual environment, make sure Django is installed; verify the installation; check my Python environment and ensure it is activated; check for multiple Python installations; check the installation path; verify the installation path; check that "models.py" is in the app directory, and the app is listed in the installed apps, etc. I have tried all of these, and the text editor cannot seem to find the module django. But it is among the files. Here is the settings, with INSTALLED APPS list: from pathlib import Path INSTALLED_APPS = [ # My apps 'learning_logs', # Default django apps 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', ]  This is the line where I get the error: from django.db import models  Unfortunately, I'm not so sure what is meant by "Related model", hence it's omission. I will say that in terminal, and error message I got was: ModuleNotFoundError: No module named 'django.utils'. I hope that makes sense.

Answer: You can create a Build System for your project where you can specify a python environment to use: { "cmd": ["/full/path/to/your/specific/python", "$file"], "selector": "source.python", "file_regex": "^\\s*File \"(...*?)\", line ([0-9]*)" }  Generally though, you don't need to run any files when working with Django. Just run python manage.py runserver (in a terminal), edit the source files and the app will be reloaded automatically; refresh the browser to see the results. If you do need to run a standalone script that uses Django, make sure to add django.setup(): import django django.setup() # now you can use models and other Django features
Question: I am trying to generate a torch vector with a specific length. I want the vector to have the same beginning elements when increasing its length using the same seed. This works when the vector's length ranges from 1 to 15 for example: For length 14 torch.manual_seed(1) torch.randn(14) tensor([ 0.6614, 0.2669, 0.0617, 0.6213, -0.4519, -0.1661, -1.5228, 0.3817, -1.0276, -0.5631, -0.8923, -0.0583, -0.1955, -0.9656])  For length 15 torch.manual_seed(1) torch.randn(15) tensor([ 0.6614, 0.2669, 0.0617, 0.6213, -0.4519, -0.1661, -1.5228, 0.3817, -1.0276, -0.5631, -0.8923, -0.0583, -0.1955, -0.9656])  But for length 16 I get a totaly different vector torch.manual_seed(1) torch.randn(16) tensor([-1.5256, -0.7502, -0.6540, -1.6095, -0.1002, -0.6092, -0.9798, -1.6091, -0.7121, 0.3037, -0.7773, -0.2515, -0.2223, 1.6871, 0.2284, 0.4676])  Can someone explain what's happennig to me and could I get a solution where the vector does not changes?

Answer: This is due to the behaviour of PRNG. Different code paths might be used. There is no guarantee that all different sequence length will produce exactly the same output samples from the PRNG. The outputs from 1-15 match while starting from 16 another (probably vectorized) code path will be used. Changes in the sequence length could dispatch to faster code paths Source: Odd result using multinomial num_samples... It seems that this may not be an issue in downgraded torch versions.
Question: Problem Description: I am trying to create a local API for my team. I think I understand broadly the mechanics of _init_.py. Let's say we have the below package structure: API/ ├── __init__.py # Top-level package init file └── core/ ├── __init__.py # Core module init file ├── calculator.py └── exceptions.py  Now if I build my API with empty _init_.py files, and then import API in my script, I won't be able to do something like: import API API.core API.core.calculator  Because the submodules have not been imported specifically. What I need to do is to add into the following into my top-level _init_.py: from . import core  And the following into my core module _init_.py: from . import calculator from . import exceptions  Now, when I do this, all imports I am making in my calculator.py or exceptions.py such as numpy or pandas are actually available through my package as follows: API.core.calculator.numpy  Question 1: What are the best practices to prevent imported libraries to show through my API ? On the same theme, let's say I want to access my calculator.py functions directly through the core keyword (let's assume _all_ variables are safely set up). Then I can add the following into my core module _init_.py from .calculator import * from .exceptions import *  which then allows me to do: API.core.my_function()  But then again, I can also call API.core.calculator.my_function() at the same time, which might be confusing for users. 2. How to prevent imported functions to be available through both my package name and module name ? I tried mixing up the approaches, but with no results, please help !

Answer: The typical best practice is to define an __all__ list in each sub-module with names you want the sub-module to export so that the parent module can import just those names with a star import. Names that you don't want exposed should be named with a leading underscore by convention so that the linters will warn the users if they try to import a "private" name. So in your example case, your API/core/calculator.py will look like: import numpy as _numpy __all__ = ['my_function'] def my_function(): ...  And then API/core/__init__.py will star-import what's exported by calculator (that is, just my_function): from .calculator import *  which then allows you to do: API.core.my_function()  but not: API.core._numpy API.core.calculator._numpy  Note that the user can always do import API.core.calculator explicitly to access _numpy in that sub-module, but that is just how the module namespace works as there is nothing really private in Python. Also note that a star import will not import names with leading underscores so if you do follow the convention of naming your "private" variables as such you don't even need to define an __all__ list. Excerpt from the documentation of More on Modules:  There is even a variant to import all names that a module defines: >>> from fibo import * fib(500) 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377  This imports all names except those beginning with an underscore (_).