eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following df: df = pd.DataFrame({"values":[1,5,7,3,0,9,8,8,7,5,8,1,0,0,0,0,2,5],"signal":['L_exit',None,None,'R_entry','R_exit',None,'L_entry','L_exit',None,'R_entry','R_exit','R_entry','R_exit','L_entry','L_exit','L_entry','R_exit',None]}) df values signal 0 1 L_exit 1 5 None 2 7 None 3 3 R_entry 4 0 R_exit 5 9 None 6 8 L_entry 7 8 L_exit 8 7 None 9 5 R_entry 10 8 R_exit 11 1 R_entry 12 0 R_exit 13 0 L_entry 14 0 L_exit 15 0 L_entry 16 2 R_exit 17 5 None  My goal is to add a tx column like this:  values signal num 0 1 L_exit nan 1 5 None nan 2 7 None nan 3 3 R_entry 1.00 4 0 R_exit 1.00 5 9 None 1.00 6 8 L_entry 1.00 7 8 L_exit 1.00 8 7 None nan 9 5 R_entry 2.00 10 8 R_exit 2.00 11 1 R_entry 2.00 12 0 R_exit 2.00 13 0 L_entry 2.00 14 0 L_exit 2.00 15 0 L_entry nan 16 2 R_exit nan 17 5 None nan  Business logic: when there's a signal of R_entry we group a tx until there's L_exit (if theres another R_entry - ignore it) visualizing  What have I tried? g = ( df['signal'].eq('R_entry') | df_tx['signal'].eq('L_exit') ).cumsum() df['tx'] = g.where(df['signal'].eq('R_entry')).groupby(g).ffill()  problem is that it increments every time it has 'R_entry'

Answer: You can first create a mask to get the contiguous R_entries up to reaching to L_exit. Then get the first R_entry per group (by comparing to the next value) and apply a cumsum. # keep only 'R_entry'/'L_exit' and get groups mask = df['signal'].where(df['signal'].isin(['R_entry', 'L_exit'])).ffill().eq('R_entry') # get groups and extend to next value (the L_exit) df['num'] = (mask.ne(mask.shift())&mask).cumsum().where(mask).ffill(limit=1)  output:  values signal num 0 1 L_exit NaN 1 5 None NaN 2 7 None NaN 3 3 R_entry 1.0 4 0 R_exit 1.0 5 9 None 1.0 6 8 L_entry 1.0 7 8 L_exit 1.0 8 7 None NaN 9 5 R_entry 2.0 10 8 R_exit 2.0 11 1 R_entry 2.0 12 0 R_exit 2.0 13 0 L_entry 2.0 14 0 L_exit 2.0 15 0 L_entry NaN 16 2 R_exit NaN 17 5 None NaN  breaking down how it works Here are the intermediate steps: df['isin+ffill'] = df['signal'].where(df['signal'].isin(['R_entry', 'L_exit'])).ffill() df['mask'] = df['isin+ffill'].eq('R_entry') df['first_of_group'] = (mask.ne(mask.shift())&mask) df['cumsum'] = df['first_of_group'].cumsum().where(mask) df['num'] = df['cumsum'].ffill(limit=1)   values signal isin+ffill mask first_of_group cumsum num 0 1 L_exit L_exit False False NaN NaN 1 5 None L_exit False False NaN NaN 2 7 None L_exit False False NaN NaN 3 3 R_entry R_entry True True 1.0 1.0 4 0 R_exit R_entry True False 1.0 1.0 5 9 None R_entry True False 1.0 1.0 6 8 L_entry R_entry True False 1.0 1.0 7 8 L_exit L_exit False False NaN 1.0 8 7 None L_exit False False NaN NaN 9 5 R_entry R_entry True True 2.0 2.0 10 8 R_exit R_entry True False 2.0 2.0 11 1 R_entry R_entry True False 2.0 2.0 12 0 R_exit R_entry True False 2.0 2.0 13 0 L_entry R_entry True False 2.0 2.0 14 0 L_exit L_exit False False NaN 2.0 15 0 L_entry L_exit False False NaN NaN 16 2 R_exit L_exit False False NaN NaN 17 5 None L_exit False False NaN NaN
Question: I have a very simple file: [Name] Streamline 1 [Data] X [ m ], Y [ m ], Z [ m ], Velocity [ m s^-1 ] 2.66747564e-01, 0.00000000e+00, 2.03140453e-01, (0.00000000e+00, 8.17744827e+00, 0.00000000e+00) 2.66958952e-01, 0.00000000e+00, 2.07407191e-01, (0.00000000e+00, 6.77392197e+00, 0.00000000e+00) 2.63460875e-01, 0.00000000e+00, 2.06593186e-01, (0.00000000e+00, 7.04168701e+00, 0.00000000e+00) 2.65424699e-01, 0.00000000e+00, 2.00831652e-01, (0.00000000e+00, 8.93691921e+00, 0.00000000e+00) 2.70607203e-01, 0.00000000e+00, 2.02286631e-01, (0.00000000e+00, 8.45830917e+00, 0.00000000e+00) 2.68299729e-01, 0.00000000e+00, 1.97365344e-01, (0.00000000e+00, 1.00771456e+01, 0.00000000e+00) ...  I need to load the velocity as a vector, into a single row. My basic code: df = pd.read_csv("C:/Users/Marek/Downloads/0deg-5ms.csv", skiprows=5)  But this attempt leads to 1st 2 cols becoming index and the rest splits into 4 columns. index_col=False can solve the issue with index, but leads to index out of range. I need a delimiter that implicitly tells pandas to ignore whatever is in brackets. I thought python ignore the separator withing brackets while reading a csv file might work but yes, I have spaces everywhere. I found some solutions that use extended functions to load files and handle them by lines, such as CSV file containing column with occasional comma in parentheses crashes pandas.read_csv and Load CSV with data surrounded by parentheses into a pandas dataframe . I however believe that this is a very easy scenario, as all lines are similar and can be solved by one-liner adding delimiter='some_regex'. I however cannot figure out, how this regex should look. It should look for delimiter , but not (.*,.*). I have tried with following, but this results in a single column: df = pd.read_csv("C:/Users/Marek/Downloads/0deg-5ms.csv", skiprows=5, delimiter=',^(\(.*,.*\))')  EDIT: got to something like this - ,|(?:(\(.*,.*\))), but this adds an empty column after each comma.

Answer: After numerous attempts, I have found an answer how to create a very simple one-liner on this. Here it is if anyone is interested: df = pd.read_csv("C:/Users/Marek/Downloads/0deg-5ms.csv", skiprows=5, delimiter=',(?![^\(]*[\)])', engine="python")  Delimiter checks for the comma in everything outside the brackets. Simple like a charm :)
Question: Suppose I have a module: mymodule/example.py: def add_one(number): return number + 1  And mymodule/__init__.py: from .example import * foo = "FOO" def bar(): return 1  Now I see the function at the root of mymodule: >>> import mymodule >>> mymodule.add_one(3) 4 >>> mymodule.foo 'FOO'  Also, I see imported add_one through dir along with example: >>> dir(mymodule) ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'add_one', 'bar', 'example', 'foo']  But when I type help(mymodule) I see only example, foo and bar, but not the imported add_one: Help on package mymodule: NAME mymodule PACKAGE CONTENTS example FUNCTIONS bar() DATA foo = 'FOO'  But I can call add_one() as the root function of mymodule. Is it possible to see it in help as root function?

Answer: From the source code of help() (docmodule under pydoc.py)  for key, value in inspect.getmembers(object, inspect.isroutine): # if __all__ exists, believe it. Otherwise use old heuristic. if (all is not None or inspect.isbuiltin(value) or inspect.getmodule(value) is object): if visiblename(key, all, object): funcs.append((key, value))  The important part is inspect.getmodule(value) is object), this is where values that are not directly part of the object itself are dropped You can add this line __all__ = ['bar', 'add_one', 'FOO']  to your __init__.py file, this way the help function will make sure to include these objects in help Keep in mind that if you do this anything you don't include in this list will not be included
Question: I wrote a Pydantic model to validate API payload. The payload has 2 attributes emailId as list and role as str { "emailId": [], "role":"Administrator" }  I need to perform two validation on attribute email -  emailId must not be empty. emailId must not contain emails from x, y, z domains.  Hence to accomplish this I wrote 2 validation methods for emailId as shown below - class PayloadValidator(BaseModel): emailId: List[str] role: str @validator("emailId") def is_email_list_empty(cls, email): if not email_id: raise ValueError("Email list is empty.") return email_id @validator("emailId") def valid_domains(cls, emailId): pass  The problem here is that if the emailId list is empty then the validators does not raise ValueError right away. It waits for all the validation method to finish execution and this is causing some serious problems to me. Is there a way I can make it happen?

Answer: If you have checks, the failure of which should interrupt the further validation, then put them in the pre=True root validator. Because field validation will not occur if pre=True root validators raise an error. For example: class PayloadValidator(BaseModel): emailId: List[str] role: str @root_validator(pre=True) def root_validate(cls, values): if not values['emailId']: raise ValueError("Email list is empty.") return values @validator("emailId") def valid_domains(cls, emailId): return emailId
Question: I'm writing unittest for a method that returns a dataframe, but, while testing the output using: self.asserEquals(mock_df, result) I'm getting ValueError: ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). Right now I'm comparing properties that serves the purpose now, self.assertEqual(mock_df.size, result.size) self.assertEqual(mock_df.col_a.to_list(), result.col_a.to_list()) self.assertEqual(mock_df.col_b.to_list(), result.col_b.to_list()) self.assertEqual(mock_df.col_c.to_list(), result.col_c.to_list())  but curious how do I assert dataframes.

Answer: import unittest import pandas as pd class TestDataFrame(unittest.TestCase): def test_dataframe(self): df1 = pd.DataFrame({'a': [1, 2], 'b': [3, 4]}) df2 = pd.DataFrame({'a': [1, 2], 'b': [3.0, 4.0]}) self.assertEqual(True, df1.equals(df2)) if __name__ == '__main__': unittest.main()
Question: I'm running into this error when trying to run a command from docker a docker container on google compute engine. Here's the stacktrace: Traceback (most recent call last): File "train.py", line 16, in <module> from stable_baselines.ppo1 import PPO1 File "/home/selfplay/.local/lib/python3.6/site-packages/stable_baselines/__init__.py", line 3, in <module> from stable_baselines.a2c import A2C File "/home/selfplay/.local/lib/python3.6/site-packages/stable_baselines/a2c/__init__.py", line 1, in <module> from stable_baselines.a2c.a2c import A2C File "/home/selfplay/.local/lib/python3.6/site-packages/stable_baselines/a2c/a2c.py", line 3, in <module> import gym File "/home/selfplay/.local/lib/python3.6/site-packages/gym/__init__.py", line 13, in <module> from gym.envs import make, spec, register File "/home/selfplay/.local/lib/python3.6/site-packages/gym/envs/__init__.py", line 10, in <module> _load_env_plugins() File "/home/selfplay/.local/lib/python3.6/site-packages/gym/envs/registration.py", line 269, in load_env_plugins context = contextlib.nullcontext() AttributeError: module 'contextlib' has no attribute 'nullcontext' 

Answer: It seems like this is an issue with python 3.6 and gym. Upgrading my container to python 3.7 fixed the issue.
Question: I've searched and I haven't found very much information on this. I'm writing a Python script to take a list of dictionaries and dump it to a yaml file. For example, I have code like the following: import yaml dict_1 = {'name' : 'name1', 'value' : 12, 'list' : [1, 2, 3], 'type' : 'doc' } dict_2 = {'name' : 'name2', 'value' : 100, 'list' : [1, 2, 3], 'type' : 'cpp' } file_info = [dict_1, dict_2] with open('test_file.yaml', 'w+') as f: yaml.dump(file_info, f)  The output that I get is: - list: - 1 - 2 - 3 name: name1 type: doc value: 12 - list: - 1 - 2 - 3 name: name2 type: cpp value: 100  When what I really want is something like this: - list: - 1 - 2 - 3 name: name1 type: doc value: 12 ## Notice the line break here - list: - 1 - 2 - 3 name: name2 type: cpp value: 100  I've tried putting \n and the end of the dictionaries, using file_info.append('\n') between the dictionaries, using None as a final key in the dictionary, but nothing has worked so far. Any help is greatly appreciated! I'm using Pyyaml 5.4.1 with Python 3.9.

Answer: You can dump each object one at a time following each with a new line. with open('test_file.yaml', 'w+') as f: for yaml_obj in file_info: f.write(yaml.dump([yaml_obj])) f.write("\n")
Question: I want to try capturing PyCharm's stop signal (when stop is pressed) in a try block, but I cannot figure out what this signal is or how to capture it in code. JetBrains doesn't provide insight into this in their documentation. I've tried catching it as BaseException but it does not seem to be an exception at all. Is this programmatically capturable at all?

Answer: I wasn't able to replicate the other answers as the stop button being sent as a keyboard interrupt. I do believe it's possible for the stop button to be implemented differently on different versions of PyCharm and OS (I'm on Linux where a different answer seems to be Windows, but I'm not positive on many aspects here) It seems to me that a kill signal is being sent, and it doesn't seem like catching that as an exception works (for me). However, I was able to somewhat catch a kill signal by referencing this post that talks about catching kill signals in Python and killing gracefully. Below is the code I used. When I press the stop button I see Hello world, but I do NOT see foobar. Also, the debugger is NOT able to be caught for me by breakpoints in handler_stop_signals by doing this, but I do see the text. So I'm not sure if this is actually going to answer your question or not, based on your needs. Also note I would never actually write code like this (using globals), but it was the simplest answer I was able to come up with. import signal import time run = True def handler_stop_signals(signum, frame): global run print("Hello world") run = False signal.signal(signal.SIGINT, handler_stop_signals) signal.signal(signal.SIGTERM, handler_stop_signals) while run: try: time.sleep(20) # do stuff including other IO stuff except: BaseException as e: print('foobar')
Question: Hi I have a file format (TSV) as like this Name type Age Weight Height Xxx M 12,34,23 50,30,60,70 4,5,6,5.5 Yxx F 21,14,32 40,50,20,40 3,4,5,5.5  I would like to add all the values in Age, Weight and Height and add a column after this, then so some percentage also, like Total_Height/Total_Weight (awk '$0=$0"\t"(NR==1?"Percentage":$8/$7)'). I have large data set and it is not possible to do with excel. Like this Name type Age Weight Height Total_Age Total_Weight Total_Height Percentage Xxx M 12,34,23 50,30,60,70 4,5,6,5.5 69 210 20.5 0.097 Yxx F 21,14,32 40,50,20,40 3,4,5,5.5 67 150 17.5 0.11 

Answer: With your shown samples please try following code. awk ' FNR==1{ print $0,"Total_Age Total_Weight Total_Height Percentage" next } FNR>1{ totAge=totWeight=totHeight=0 split($3,tmp,",") for(i in tmp){ totAge+=tmp[i] } split($4,tmp,",") for(i in tmp){ totWeight+=tmp[i] } split($5,tmp,",") for(i in tmp){ totHeight+=tmp[i] } $(NF+1)=totAge $(NF+1)=totWeight $(NF+1)=totHeight $(NF+1)=$(NF-1)==0?"N/A":$NF/$(NF-1) } 1' Input_file | column -t  OR adding a bit short version of above awk code: awk ' BEGIN{OFS="\t"} FNR==1{ print $0,"Total_Age Total_Weight Total_Height Percentage" next } FNR>1{ totAge=totWeight=totHeight=0 split($3,tmp,",") for(i in tmp){ totAge+=tmp[i] } split($4,tmp,",") for(i in tmp){ totWeight+=tmp[i] } split($5,tmp,",") for(i in tmp){ totHeight+=tmp[i] } $(NF+1)=totAge OFS totWeight OFS totHeight $0=$0 $(NF+1)=( $(NF-1)==0 ? "N/A" : $NF/$(NF-1) ) } 1' Input_file | column -t  Explanation: Simple explanation would be, take sum of 3rd, 4th and 5th columns and assign them to last column of line. Accordingly add column value which has divide value of last and 2nd last columns as per OP's request. Using column -t to make it look better on output.
Question: I'm trying to understand what is this variable called context in Airflow operators. as example: def execute(self, **context**).  Where it comes from? where can I set it? when and how can I use it inside my function? Another question is What is *context and **context? I saw few examples that uses this variable like this: def execute(self, *context) / def execute(self, **context).  What is the difference and when should I use *context and **context

Answer: When Airflow runs a task, it collects several variables and passes these to the context argument on the execute() method. These variables hold information about the current task, you can find the list here: https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html#default-variables. Information from the context can be used in your task, for example to reference a folder yyyymmdd, where the date is fetched from the variable ds_nodash, a variable in the context: def do_stuff(**context): data_path = f"/path/to/data/{context['ds_nodash']}" # write file to data_path... PythonOperator(task_id="do_stuff", python_callable=do_stuff)  *context and **context are different Python notations for accepting arguments in a function. Google for "args vs kwargs" to find more on this topic. Basically *context accepts non-keyword arguments, while **context takes keyword arguments: def print_context(*context_nokeywords, **context_keywords): print(f"Non keywords args: {context_nokeywords}") print(f"Keywords args: {context_keywords}") print_context("a", "b", "c", a="1", b="2", c="3") # Non keywords args: ('a', 'b', 'c') # Keywords args: {'a': '1', 'b': '2', 'c': '3'}
Question: I want to download the GPT-2 model and tokeniser. For open-end generation, HuggingFace sets the padding token ID to be equal to the end-of-sentence token ID, so I configured it manually using :  import tensorflow as tf from transformers import TFGPT2LMHeadModel, GPT2Tokenizer tokenizer = GPT2Tokenizer.from_pretrained("gpt2") model = TFGPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)  However, it gives me the following error:  TypeError: ('Keyword argument not understood:', 'pad_token_id')  I haven't been able to find a solution for this nor do I understand why I am getting this error. Insights will be appreciated.

Answer: Your code does not throw any error for me - I would try re-installing the most recent version of transformers - if that is a viable solution for you.
Question: I am creating a histrogram (frecuency vs. count) and I want to add kernel density estimate line in a different colour. How can I do this? I want to change the colour for example sns.histplot(data=penguins, x="flipper_length_mm", kde=True)  Example taken from https://seaborn.pydata.org/generated/seaborn.histplot.html

Answer: histplot's line_kws={...} is meant to change the appearance of the kde line. However, the current seaborn version doesn't allow changing the color that way, probably because the color goes together with the hue parameter (although hue isn't used in this case). import seaborn as sns penguins = sns.load_dataset('penguins') ax = sns.histplot(data=penguins, x="flipper_length_mm", kde=True, line_kws={'color': 'crimson', 'lw': 5, 'ls': ':'})   In seaborn's github, it is suggested to draw the histplot and the kdeplot separately. For both to match in the y-direction, it is necessary to use histplot with stat='density' (the kdeplot doesn't have a parameter to use histplot's default stat='count'). penguins = sns.load_dataset('penguins') ax = sns.histplot(data=penguins, x="flipper_length_mm", kde=False, stat='density') sns.kdeplot(data=penguins, x="flipper_length_mm", color='crimson', ax=ax)   If the count statistics is really needed, an alternative is to change the line color via matplotlib: penguins = sns.load_dataset('penguins') ax = sns.histplot(data=penguins, x="flipper_length_mm", kde=True) ax.lines[0].set_color('crimson')
Question: When working with the built in decimal module in python I can round decimals as follows. Decimal(50.212345).quantize(Decimal('0.01')) > Decimal('50.21')  But I can also round the same number with the built in round function round(Decimal(50.212345), 2) > Decimal('50.21')  Why would I use one instead of the other when rounding Decimals? In previous answers about rounding decimals, users suggested to use quantize because the built in round function would return a value of type float. Based on my testing, these both return a Decimal. Other than syntax, is there a reason to choose one over the other?

Answer: The return types aren't always the same. round() used with a single argument actually returns an int: >>> round(5.3) 5 >>> round(decimal.Decimal("5.3")) 5  Other than that, suit yourself. quantize() is especially handy if you want a deoimal rounded to "the same" precision as another decimal you already have. >>> x = decimal.Decimal("123.456") >>> x*x Decimal('15241.383936') >>> (x*x).quantize(x) Decimal('15241.384')  See? The code doing this doesn't have to know that x originally had 3 digits after the decimal point. Just passing x to quantize() forces the function to round back to the same precision as the original x, regardless of what that may be. quantize() is also necessary if you want to use a rounding mode other than the default nearest/even. >>> (x*x).quantize(x, decimal.ROUND_FLOOR) Decimal('15241.383')
Question: I have a function called from an async function without await, and my function needs to call async functions. I can do this with asyncio.get_running_loop().create_task(sleep()) but the run_until_complete at the top level doesn't run until the new task is complete. How do I get the event loop to run until the new task is complete? I can't make my function async because it's not called with await. I can't change future or sleep. I'm only in control of in_control. import asyncio def in_control(sleep): """ How do I get this to run until complete? """ return asyncio.get_running_loop().create_task(sleep()) async def future(): async def sleep(): await asyncio.sleep(10) print('ok') in_control(sleep) asyncio.get_event_loop().run_until_complete(future()) 

Answer: It appears that the package nest_asyncio will help you out here. I've also included in the example fetching the return value of the task. import asyncio import nest_asyncio def in_control(sleep): print("In control") nest_asyncio.apply() loop = asyncio.get_running_loop() task = loop.create_task(sleep()) loop.run_until_complete(task) print(task.result()) return async def future(): async def sleep(): for timer in range(10): print(timer) await asyncio.sleep(1) print("Sleep finished") return "Sleep return" in_control(sleep) print("Out of control") asyncio.get_event_loop().run_until_complete(future())  Result: In control 0 1 2 3 4 5 6 7 8 9 Sleep finished Sleep return Out of control [Finished in 10.2s]
Question: I have a functions in Python which has a series of if statements. def some_func(): if condition_1: return result_1 if condition_2: return result_2 ... #other if statements  But I want that the order of these if statements is changed every time I call the function. Like if I call the function there can be a case when condition_1 and condition_2 both are true but since condition_1 is checked first thus I will get result_1 but I want that when I call the function second time some other random condition to be checked. (since I have only five if statements in the function therefore any one of the five conditions). So is there any way to do this like storing the conditions in a list or something. Any help will be appreciated.

Answer: You can create list of condition and result then shuffle them like below: import random def some_func(x): lst_condition_result = [((x>1), True), ((x>2), False)] random.shuffle(lst_condition_result) for condition, result in lst_condition_result: if condition: return result  Output: >>> some_func(20) True >>> some_func(20) False >>> some_func(20) False
Question: I have a router that fetches all data from the database. Here is my code: @router.get('/articles/', response_model=List[articles_schema.Articles]) async def main_endpoint(): query = articles_model.articles.select().where(articles_model.articles.c.status == 2) return await db.database.fetch_all(query)  The response is an array that contains JSON objects like this [ { "title": "example1", "content": "example_content1" }, { "title": "example2", "content": "example_content2" }, ]  But I want to make the response like this: { "items": [ { "title": "example1", "content": "example_content1" }, { "title": "example2", "content": "example_content2" }, ] }  How can I achieve that? Please help. Thank you in advance

Answer: You could simply define another model containing the items list as a field: from pydantic import BaseModel from typing import List class ResponseModel(BaseModel): items: List[articles_schema.Articles]  and use it in the response: @router.get('/articles/', response_model=ResponseModel) async def main_endpoint(): query = articles_model.articles.select().where( articles_model.articles.c.status == 2 ) return ResponseModel( items=await db.database.fetch_all(query), )