eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Could you please suggest is there any way to keep all the repeatable (duplicate) keys by adding prefix or suffix. In the below example, the address key is duplicated 3 times. It may vary (1 to 3 times). I want to get the output as in the expected output with adding a suffix to make the key unique. Currently the update function is overwriting the key value. list = ['name:John','age:25','Address:Chicago','Address:Phoenix','Address:Washington','email:John@email.com'] dic = {} for i in list: j=i.split(':') dic.update({j[0]:j[1]}) print(dic)  Current output: {'name': 'John', 'age': '25', 'Address': 'Washington', 'email': 'John@email.com'} Expected output: {'name': 'John', 'age': '25', 'Address1': 'Chicago', 'Address2': 'Phoenix', 'Address3': 'Washington', 'email': 'John@email.com'} Tried the below: list = ['name:John','age:25','Address:Chicago','Address:Phoenix','Address:Washington','email:John@email.com'] dic = {} for i in list: j=i.split(':') dic.update({j[0]:j[1]}) print(dic)  Expected output: {'name': 'John', 'age': '25', 'Address1': 'Chicago', 'Address2': 'Phoenix', 'Address3': 'Washington', 'email': 'John@email.com'}

Answer: You can use something like this: list_ = ['name:John','age:25','Address:Chicago','Address:Phoenix','Address:Washington','email:John@email.com'] dic = {} for i in list_: j = i.split(':') key_ = j[0] count = 0 # counts the number of duplicates while key_ in dic: count += 1 key_ = j[0] + str(count) dic[key_] = j[1]  Output: {'name': 'John', 'age': '25', 'Address': 'Chicago', 'Address1': 'Phoenix', 'Address2': 'Washington', 'email': 'John@email.com'}  PS. don't use the python keyword list to name your variables as it overrides the type list
Question: My list consists on several items: ["book","rule","eraser","clipboard","pencil",etc]  Let say I have an element ["book"] and I want to match with another element ["rule"] with the same len (4). then I want to remove both elements from the list. ["eraser","clipboard","pencil",etc]  I tried using a for loop and zip(lists, lists[1:]) but I can only remove the elements next to each other while other elements (like papers and pencil, len = 6) are never removed. I want to remove same length words and have a final list with only the words whose length has no match or found no pair. For example: ["book","rule","eraser","clipboard","pencil","book"] then the final list will be: ["clipboard","book"]  as book and clipboard found no pair.

Answer: Much simpler, think of it as turning on or off a flag. If the flag is on, you turn it off by deleting. If it's off, you turn it on by adding the word. Flags are based on same length: lst = ["book", "rule", "eraser", "clipboard", "pencil", "book"] output = {} for item in lst: length = len(item) popped = output.pop(length, None) if popped is None: output[length] = item print(list(output.values()))
Question: I enjoy using type hinting (annotation) and have used it for some time to help write clean code. I appreciate that they are just hints and as such do not affect the code. But today I saw a video where the linter picked up the hint with a warning (a squiggly yellow underline), which looks really helpful. My VS Code does not pick this up in the linter. Here is an image of what I expect (with annotations):  So my question is, how can I achieve this? For example, is there a specific linter or setting that would do this?

Answer: If you are using Pylance, you can add a new line to your settings.json (you need to restart VS Code after updating the file): "python.analysis.typeCheckingMode": "basic"  The default value is off, the other possible values are basic and strict. The following screenshot shows warnings for different situations: wrong variable type, wrong function return type and wrong parameter type.
Question: For some reason FastAPI doesn't respond to any requests while a request is handled. I would expect FastAPI to be able to handle multiple requests at the same time. I would like to allow multiple calls at the same time as multiple users might be accessing the REST API. Minimal Example: Asynchronous Processes After starting the server: uvicorn minimal:app --reload, running the request_test run request_test and executing test(), I get as expected {'message': 'Done'}. However, when I execute it again within the 20 second frame of the first request, the request is not being processed until the sleep_async from the first call is finished. Without asynchronous Processes The same problem (that I describe below) exists even if I don't use asynchronous calls and wait directly within async def info. That doesn't make sense to me. FastAPI: minimal #!/usr/bin/env python3 from fastapi import FastAPI from fastapi.responses import JSONResponse import time import asyncio app = FastAPI() @app.get("/test/info/") async def info(): async def sleep_async(): time.sleep(20) print("Task completed!") asyncio.create_task(sleep_async()) return JSONResponse(content={"message": "Done"})  Test: request_test #!/usr/bin/env python3 import requests def test(): print("Before") response = requests.get(f"http://localhost:8000/test/info") print("After") response_data = response.json() print(response_data) 

Answer: You need to set the number of workers as shown in the uvicorn settings --workers <int>
Question: I have the following class in a FastAPI application: import asyncio import logging from multiprocessing import Lock, Process from .production_status import Job as ProductionStatusJob class JobScheduler: loop = None logger = logging.getLogger("job_scheduler") process_lock = Lock() JOBS = [ProductionStatusJob] @classmethod def start(cls) -> None: cls.logger.info("Starting Up (1/2)") Process(target=cls._loop).start() @classmethod def _loop(cls) -> None: cls.loop = asyncio.get_event_loop() cls.loop.create_task(cls._run()) cls.logger.info("Startup Complete (2/2)") cls.loop.run_forever() cls.loop.close() @classmethod async def _run(cls) -> None: while True: ... @classmethod async def stop(cls) -> None: cls.logger.info("Shutting Down (1/2)") with cls.process_lock: cls.loop.stop() # <= This Line cls.loop.close() cls.logger.info("Shutdown Complete (2/2)") cls.loop = None  On the startup and shutdown events of the FastAPI application, the JobScheduler.start() and JobScheduler.stop() methods will be called. The start method works smoothly, but in stop I get an error: File "/backend/app/main.py", line 146, in stop_job_scheduler 2023-08-16 11:46:27 await job_scheduler.stop() 2023-08-16 11:46:27 File "/backend/app/jobs/__init__.py", line 59, in stop 2023-08-16 11:46:27 cls.loop.stop() 2023-08-16 11:46:27 AttributeError: 'NoneType' object has no attribute 'stop'  But cls.loop is set during the _loop method (which is executed at the end of start) - so why does cls.loop still have its initial None value when the stop method is called? Are there any better approaches to clean up the background processes when the FastAPI application calls shutdown?

Answer: multiprocessing in Python is funny. It's more powerful than multithreading but also comes with some caveats. The first of those is that you're actually running a different Python interpreter entirely. That means that global variables and the like are going to get a new copy for each process you run. Depending on your operating system and choice of start method, your processes may be forked or spawned. A spawned process will start anew, as though a new Python program was just spun up. A forked process will get all of the current values of variables from the source process, but it'll still copy all of those variables. Future changes to either process will not affect the other, without explicit synchronization using one of the multiprocessing helpers. You can use a Manager to synchronize data between processes explicitly. This acts sort of like a local server that both processes connect to. For more explicitly pub-sub data, you can also use a Queue to pass information from one process to another.
Question: Is there a way to rewrite this: import numpy import polars df = (polars .DataFrame(dict( j=numpy.random.randint(10, 99, 20), )) .with_row_index() .select( g=polars.col('index') // 3, j='j' ) .with_columns(rn=1) .with_columns( rn=polars.col('rn').shift().fill_null(0).cum_sum().over('g') ) ) print(df) g (u32) j (i64) rn (i32) 0 47 0 0 22 1 0 82 2 1 19 0 1 85 1 1 15 2 2 89 0 2 74 1 2 26 2 3 11 0 3 86 1 3 81 2 4 16 0 4 35 1 4 60 2 5 30 0 5 28 1 5 94 2 6 21 0 6 38 1 shape: (20, 3)  so it adds rn column without requiring it to add a column full of 1s first? I.e. somehow rewrite this part:  .with_columns(rn=1) .with_columns( rn=polars.col('rn').shift().fill_null(0).cum_sum().over('g') )  so that:  .with_columns(rn=1)  is not required? Basically reduce two expressions to one. Or any other / better way to add a row count per group?

Answer: It can be done by generating an .int_range() using the length of each group. df.with_columns(rn = pl.int_range(pl.len()).over("g"))  shape: (20, 3) ┌─────┬─────┬─────┐ │ g ┆ j ┆ rn │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 0 ┆ 14 ┆ 0 │ # group_len = 3, range = [0, 1, 2] │ 0 ┆ 81 ┆ 1 │ │ 0 ┆ 72 ┆ 2 │ │ 1 ┆ 34 ┆ 0 │ │ 1 ┆ 90 ┆ 1 │ │ … ┆ … ┆ … │ │ 5 ┆ 26 ┆ 0 │ │ 5 ┆ 44 ┆ 1 │ │ 5 ┆ 27 ┆ 2 │ │ 6 ┆ 70 ┆ 0 │ │ 6 ┆ 86 ┆ 1 │ └─────┴─────┴─────┘
Question: I noticed a thing in python polars. I’m not sure but seems that pl.when().then().otherwise() is slow somewhere. For instance, for dataframe: df = pl.DataFrame({ 'A': [randint(1, 10**15) for _ in range(30_000_000)], 'B': [randint(1, 10**15) for _ in range(30_000_000)], }, schema={ 'A': pl.UInt64, 'B': pl.UInt64, })  Horizontal min with pl.min_horizontal: df.with_columns( pl.min_horizontal(['A', 'B']).alias('min_column') ) 92.4 ms ± 16.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  And the same with when().then().otherwise(): df.with_columns( pl.when( pl.col('A') < pl.col('B') ).then(pl.col('A')).otherwise(pl.col('B')).alias('min_column'), ) 458 ms ± 75.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  I measure explicitly the when part and seems that it is not a bottleneck. df.with_columns((pl.col('A') < pl.col('B')).alias('column_comparison')) 49.2 ms ± 6.23 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  If remove otherwise() it will be even slower. df.with_columns( pl.when( pl.col('A') < pl.col('B') ).then(pl.col('A')).alias('min_column') ) 664 ms ± 19.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  I also have tried some other methods for horizontal reducing such as pl.reduce or pl.fold and seems that they all are much faster than when().then(). So the questions here:  Is it expected behavior? Why pl.when().then() is much slower than other expressions? In which cases should we avoid when().then().otherwise()? 

Answer: I've got some comments from Polars developers at discord.  I don't see anything out of the ordinary here. Removing otherwise just means .otherwise(pl.lit(None)) is called in the background. It will have to create that column rather than using the existing one. So it will be slower. If you can write your expression as a fold it might be faster, as you have noticed with min_horizontal.  So my conclusion here: when you have a task to reduce several columns into one column, it is better choice to use fold or reduce methods when possible, instead of when().then(). EDIT Sinse polars 0.20.17 there is huge speed up in when then operations, caused by refactoring if-then-else kernels. Benchmarks: https://github.com/pola-rs/polars/pull/15131 So now it is not an issue to use if-then-else if it is necessary.
Question: I am changing some of our code from Azure ML's SDK v1 to v2. However, when I invoke pipelines with components via ml_client.jobs.create_or_update, I just can't get them to use my environment variables. Here is what I am doing: preprocessing_component = load_component( source=Path(__file__).parent / "preprocessing_component.yaml" ) @pipeline() def example_train_pipeline(input_data_path): preprocess_step = preprocessing_component( input_data_path=input_data_path pipeline_job = example_train_pipeline( input_data_path=Input( type=AssetTypes.URI_FILE, path="xxx", ) ) pipeline_job.settings.default_compute = e.cluster_name pipeline_job = ml_client.jobs.create_or_update( pipeline_job, experiment_name=experiment_name )  I tried to set .env_variables when creating my AZ ML environment (which is loaded for this pipeline's component in the yaml). This stated this parameters was deprecated and I should use RunConfig.environment_variables instead. Thing is, I can't find docs on how to use a RunConfig with ml_client.jobs.create_or_update. I tried just passing a RunConfig with variables set via run_config.environment_variables to create_or_update, but this had no apparrent effect.

Answer: With the introduction of Azure ML SDK v2, the concept of components has been emphasized. These components allow you to define specific environments individually. You can set environment variables for each component as following sample code. You can define environment variables using Python code: environment_variables = {"environ": "val"} command_function = command( display_name="command-function-job", environment=environment, command='echo "hello world"', distribution=distribution, resources=resources, environment_variables=environment_variables, inputs=inputs, outputs=outputs, )
Question: I recently installed Anaconda and Python (3.11.4). I created my env in Anaconda, and when I tried to install rasa (with "pip install rasa") conda showed me this error. Please help me ERROR when i installed rasa pip  Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "C:\Users\Eliseo\AppData\Local\Temp\pip-install-qo_63x3c\absl-py_16e4b8fcbea9469084883112126934fa\setup.py", line 34, in <module> raise RuntimeError('Python version 2.7 or 3.4+ is required.') RuntimeError: Python version 2.7 or 3.4+ is required. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.  I tried uninstalling Python, and "conda clean --all" in Anaconda Prompt. But it doesn't work.

Answer: In the official Rasa documentation, it says that it is not compactible with modern Python versions. The most current and 100% compact is installing python3.8 wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz tar -xf Python-3.8.0.tar.xz cd Python-3.8.0 ./configure --enable-optimizations make sudo make altinstall  And to create your application.(example on tmp path) cd /tmp/myapp python3.8 -m venv venv source venv/bin/activate pip install rasa source venv/bin/activate  Attention, you may get an outdated pip error pip install --upgrade pip
Question: Recently I keep running into problems with my python notebooks in vscode where vscode doesn't see the installed ipykernel. There are several posts on this issue with suggestions to update certain packages (VSCode not picking up ipykernel, Python requires ipykernel to be installed, vscode not detecting ipykernel, verified it is actually installed, Install ipykernel in vscode - ipynb (Jupyter), ...) This makes me wonder what the actual minimum requirements are. Different "offical" channels mention different dependencies:  https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_setting-up-your-environment mentions jupyter https://github.com/microsoft/vscode-jupyter/wiki/Jupyter-Kernels-and-the-Jupyter-Extension#python-environments mentions ipython and ipykernel https://github.com/microsoft/vscode-jupyter/wiki/Installing-Jupyter mentions jupyter may be required.  Previously I only had ipykernel notebook  installed in the conda environment which worked just fine. So what are the actual requirements to run jupyter notebooks in vscode? What are the needed packages with versions?

Answer: The minimum requirement according to the ticket https://github.com/microsoft/vscode-jupyter/issues/14130 is ipykernel  The jupyter package is only required when asked to install. It has to do with native zmq modules not working on the platform.
Question: In Python (3.11) why does the use of an assignment expression (the "walrus operator") require wrapping in parentheses when used inside an f-string? For example: #!/usr/bin/env python from pathlib import Path import torch DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") ckpt_dir = Path("/home/path/to/checkpoints") _ckpt = next(ckpt_dir.iterdir()) print(_ckpt) sdict = torch.load(_ckpt, map_location=DEVICE) model_dict = sdict["state_dict"] for k, v in model_dict.items(): print(k) print(type(v)) print(_shape := v.size()) print(f"{(_numel := v.numel())}") print(_numel == torch.prod(torch.tensor(_shape)))  The code block above with print(f"{_numel := v.numel()}") instead does not parse. What about the parsing / AST creation mandates this?

Answer: This behavior was explicitly specified in the original PEP for the assignment expressions (aka the walrus operator). The reason for this was to preserve backward compatibility with formatted string literals. Before assignment expressions were added, you could already write f-strings like f"{x:=y}", which meant "format x using the format specification =y". Quoting PEP 572 – Assignment Expressions:  Assignment expressions inside of f-strings require parentheses. Example: >>> f'{(x:=10)}' # Valid, uses assignment expression '10' >>> x = 10 >>> f'{x:=10}' # Valid, passes '=10' to formatter ' 10'  This shows that what looks like an assignment operator in an f-string is not always an assignment operator. The f-string parser uses : to indicate formatting options. To preserve backward compatibility, assignment operator usage inside of f-strings must be parenthesized. As noted above, this usage of the assignment operator is not recommended.
Question: What is the recommended way to validate the access token in backend? Any library that handles it? Another team has implemented the frontend they send the access token in the Bearer attributed in the header. I found https://github.com/odwyersoftware/azure-ad-verify-token but it has only 17 Stars. I thought microsoft should have support for it in MSAL (https://github.com/AzureAD/microsoft-authentication-library-for-python) but seems not. Any suggestions on how to implement it in a secure way? Or any good libs that handles the validation. I have tried write the code my self but I get problems but worried its not secured and the code got messy. Also tried above lib but should like to have some more popular so its not a security risk.

Answer: Microsoft does not have a Python library to validate access tokens. Nevertheless, I found this official sample. You can check the requires_auth() function, which is used to validate the access token.
Question: For instance, I'd like to convert "91234 5g556７\t7₇89^" into ["9","1","2","3","4 5g55","6７\t7₇8","9^"]. Of course this can be done in a for loop without using any regular expressions, but I want to know if this can be done via a singular regular expression. At present I find two ways to do so: >>> import re >>> def way0(char: str): ... delimiter = "" ... while True: ... delimiter += " " ... if delimiter not in char: ... substitution = re.compile("([0-9])(?!\\1)([0-9])") ... replacement = "\\1"+delimiter+"\\2" ... cin = [char] ... while True: ... cout = [] ... for term in cin: cout.extend(substitution.sub(replacement,term).split(delimiter)) ... if cout == cin: ... return cin ... else: ... cin = cout ... >>> way0("91234 5g556７\t7₇89^") ['9', '1', '2', '3', '4 5g55', '6７\t7₇8', '9^'] >>> import functools >>> way1 = lambda w: ["".join(list(y)) for x, y in itertools.groupby(re.split("(0+|1+|2+|3+|4+|5+|6+|7+|8+|9+)", w), lambda z: z != "") if x] >>> way1("91234 5g556７\t7₇89^") ['9', '1', '2', '3', '4 5g55', '6７\t7₇8', '9^']  However, neither way0 nor way1 is concise (and ideal). I have read the help page of re.split; unfortunately, the following code does not return the desired output: >>> re.split(r"(\d)(?!\1)(\d)","91234 5g556７\t7₇89^") ['', '9', '1', '', '2', '3', '4 5g5', '5', '6', '７\t7₇', '8', '9', '^']  Can re.split solve this problem directly (that is, without extra conversions)? (Note that here I don't focus on the efficiency.) There are some questions of this topic before (for example, Regular expression of two digit number where two digits are not same, Regex to match 2 digit but different numbers, and Regular expression to match sets of numbers that are not equal nor reversed), but they are about "RegMatch". In fact, my question is about "RegSplit" (rather than "RegMatch" or "RegReplace").

Answer: If you want to solve this using re.split without capturing and any further processing in one step, an idea is to use only lookarounds and in the lookbehind disallow two same digits looking ahead. (?=[0-9])(?<=(?!00|11|22|33|44|55|66|77|88|99)[0-9])  See this demo at regex101 or the Python demo at tio.run The way it works is obvious. The lookarounds find any position between two digits. Inside the lookbehind the negative lookahead prevents matching (before) if two same digits are ahead. I used [0-9] and not \d because unsure if \d matches unicode digits in your Python version.
Question: I'm experimenting with LangChain's AgentType.CHAT_ZERO_SHOT_REACT agent. By its name I'd assume this is an agent intended for chat use and I've given it memory but it doesn't seem able to access its memory. What else do I need to do so that this will access its memory? Or have I incorrectly assumed that this agent can handle chats? Here is my code and sample output: llm = ChatOpenAI(model_name="gpt-4", temperature=0) tools = load_tools(["llm-math", "wolfram-alpha", "wikipedia"], llm=llm) memory = ConversationBufferMemory(memory_key="chat_history") agent_test = initialize_agent( tools=tools, llm=llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, handle_parsing_errors=True, memory=memory, verbose=True )  >>> agent_test.run("What is the height of the empire state building?") 'The Empire State Building stands a total of 1,454 feet tall, including its antenna.' >>> agent_test.run("What was the last question I asked?") "I'm sorry, but I can't provide the information you're looking for." 

Answer: It does not. that's indicated by zero-shot which means just look at the current prompt. from here  Zero-shot means the agent functions on the current action only — it has no memory. It uses the ReAct framework to decide which tool to use, based solely on the tool’s description.  I think when you work with this agent type, you should add description to the tool. so that based on the description, llm will infer which tool used. that is the description part of ""zero-shot-react-description". example from same link above: math_tool = Tool( name='Calculator', func=llm_math.run, description='Useful for when you need to answer questions about math.' )  When the llm sees the prompt, if it infers that prompt is related to math, it will use the math_tool If you want to use memory, you should be using chat-conversational-react-description  from langchain.memory import ConversationBufferMemory from langchain.chat_models import ChatOpenAI memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True) llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY, temperature=0) agent_chain = initialize_agent(tools, llm, agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory)