eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So I was upgrading DAGs from airflow version 1.12.15 to 2.2.2 and DOWNGRADING python from 3.8 to 3.7 (since MWAA doesn't support python 3.8). The DAG is working fine on the previous setup but shows this error on the MWAA setup: Broken DAG: [/usr/local/airflow/dags/google_analytics_import.py] Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1474, in set_downstream self._set_relatives(task_or_task_list, upstream=False, edge_modifier=edge_modifier) File "/usr/local/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1412, in _set_relatives task_object.update_relative(self, not upstream) AttributeError: 'DAG' object has no attribute 'update_relative'  This is the built-in function that seems to be failing:  def set_downstream( self, task_or_task_list: Union[TaskMixin, Sequence[TaskMixin]], edge_modifier: Optional[EdgeModifier] = None, ) -> None: """ Set a task or a task list to be directly downstream from the current task. Required by TaskMixin. """ self._set_relatives(task_or_task_list, upstream=False, edge_modifier=edge_modifier)  There is the code we are trying to run in the DAG:  for report in reports: dag << PythonOperator( task_id=f"task_{report}", python_callable=process, op_kwargs={ "conn": "snowflake_production", "table": report, }, provide_context=True, )  I am thinking this transition from Python 3.8 to 3.7 is causing this issue but I am not sure. Did anyone run across a similar issue ?

Answer: For Airflow>=2.0.0 Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported. Trying to do: dag = DAG("my_dag") dummy = DummyOperator(task_id="dummy") dag >> dummy  Will not work. Dependencies should be set only between operators. You should use context manager: with DAG("my_dag") as dag: dummy = DummyOperator(task_id="dummy")  It already handles the relations of operator to DAG object. If you prefer not to, then use the dag parameter in the operator constructor as: DummyOperator(task_id="dummy", dag=dag)
Question: I am trying to fine-tune the BERT language model on my own data. I've gone through their docs, but their tasks seem to be not quite what I need, since my end goal is embedding text. Here's my code: from datasets import load_dataset from transformers import BertTokenizerFast, AutoModel, TrainingArguments, Trainer import glob import os base_path = '../data/' model_name = 'bert-base-uncased' max_length = 512 checkpoints_dir = 'checkpoints' tokenizer = BertTokenizerFast.from_pretrained(model_name, do_lower_case=True) def tokenize_function(examples): return tokenizer(examples['text'], padding=True, truncation=True, max_length=max_length) dataset = load_dataset('text', data_files={ 'train': f'{base_path}train.txt', 'test': f'{base_path}test.txt', 'validation': f'{base_path}valid.txt' } ) print('Tokenizing data. This may take a while...') tokenized_dataset = dataset.map(tokenize_function, batched=True) train_dataset = tokenized_dataset['train'] eval_dataset = tokenized_dataset['test'] model = AutoModel.from_pretrained(model_name) training_args = TrainingArguments(checkpoints_dir) print('Training the model...') trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) trainer.train()  I get the following error:  File "train_lm_hf.py", line 44, in <module> trainer.train() ... File "/opt/conda/lib/python3.7/site-packages/transformers/data/data_collator.py", line 130, in torch_default_data_collator batch[k] = torch.tensor([f[k] for f in features]) ValueError: expected sequence of length 165 at dim 1 (got 128)  What am I doing wrong?

Answer: I fixed this solution by changing the tokenize function to: def tokenize_function(examples): return tokenizer(examples['text'], padding='max_length', truncation=True, max_length=max_length)  (note the padding argument). Also, I used a data collator like so: data_collator = DataCollatorForLanguageModeling( tokenizer=tokenizer, mlm=True, mlm_probability=0.15 ) trainer = Trainer( model=model, args=training_args, data_collator=data_collator, train_dataset=train_dataset, eval_dataset=eval_dataset )
Question: I am very new to blockchain programming and programming in general. I want to generate my SOL address using the mnemonic seed phrase with the derivation path "m/44'/501'/0'/0". I can't find a proper BIP44 module for python where you can specify the derivation path.

Answer: After a long search through the internet, I have finally found a way of solving my problem that I want to share with you. from bip_utils import * MNEMONIC = "...12 words phrase..." seed_bytes = Bip39SeedGenerator(MNEMONIC).Generate("") bip44_mst_ctx = Bip44.FromSeed(seed_bytes, Bip44Coins.SOLANA) bip44_acc_ctx = bip44_mst_ctx.Purpose().Coin().Account(0) bip44_chg_ctx = bip44_acc_ctx.Change(Bip44Changes.CHAIN_EXT) print(bip44_chg_ctx.PublicKey().ToAddress())  This code outputs the first address of your mnemonic. This is only for Sollet and Phantom wallet! If you are using Solflare you can cut the line bip44_chg_ctx = bip44_acc_ctx.Change(Bip44Changes.CHAIN_EXT) out!
Question: I'm trying to define a decorator in order to execute a class method, try it first and, if an error is detected, raise it mentioning the method in which failed, so as to the user could see in which method is the error. Here I show a MRE (Minimal, Reproducible Example) of my code. from functools import wraps def trier(func): """Decorator for trying A-class methods""" @wraps(func) def inner_func(self, name, *args): try: func(self, *args) except: print(f"An error apeared while {name}") return inner_func class A: def __init__(self): self._animals = 2 self._humans = 5 @trier('getting animals') def animals(self, num): return self._animals + num @trier('getting humans') def humans(self): return self._humans A().animals  Many errors are raising, like:  TypeError: inner_func() missing 1 required positional argument: 'name'  or misunderstanding self class with self function.

Answer: As an alternative to Stefan's answer, the following simply uses @trier without any parameters to decorate functions, and then when printing out the error message we can get the name with func.__name__. from functools import wraps def trier(func): """Decorator for trying A-class methods""" @wraps(func) def inner_func(self, *args, **kwargs): try: return func(self, *args, **kwargs) except: print(f"An error apeared in {func.__name__}") return inner_func class A: def __init__(self): self._animals = 2 self._humans = 5 @trier def animals(self, num): return self._animals + num @trier def humans(self): return self._humans print(A().animals(1))  I also fixed a couple of bugs in the code: In trier's try and except the result of calling func was never returned, and you need to include **kwargs in addition to *args so you can use named parameters. I.e. A().animals(num=1) only works when you handle kwargs.
Question: I'm having difficulties applying customs functions to a groupby operation in pandas. Let's suppose that I have the following DataFrame to work with: import pandas as pd df = pd.DataFrame( { "id": [1, 1, 2, 2], "flag": ["A", "A", "B", "B"], "value1": [520, 250, 180, 360], "value2": [11, 5, 7, 2], } ) print(df) id flag value1 value2 0 1 A 520 11 1 1 A 250 5 2 2 B 180 7 3 2 B 360 2  I need to apply 4 aggregate functions to the above DataFrame grouped by id and flag. Specifically, for each id and flag:  Calculate the mean of value1; Calculate the sum of value2; Calculate the mean of (value1 * value2) / 12; Calculate the sum of (value1 / value2).  I don't have any issues with the first two. This is what I did to calculate them: df.groupby(["id", "flag"]).agg({"value1": ["mean"], "value2": ["sum"]}) value1 value2 mean sum id flag 1 A 385.0 16 2 B 270.0 9  My problems are related to the last two aggregates. I search here for similar problems and I think I need to create two custom functions and apply them to the groupby object. Unfortunately, all my attempts failed and I wasn't able to work this out. Also, if possible, I want all results to be in a single DataFrame like below (hopefully, I've calculated the numbers correctly):  value1 value2 mean sum func1 func2 id flag 1 A 385.0 16 290.42 97.27 2 B 270.0 9 82.5 205.71 

Answer: groupby().agg. only takes in values of one columns. With custom functions involving several columns, I would do something like this: groupby = df.groupby(['id','flag']) out = pd.DataFrame({ 'value1': groupby['value1'].mean(), 'value2': groupby['value2'].sum(), 'value3': groupby.apply(lambda x: (x['value1'] * x['value2']).mean()/12), 'value4': groupby.apply(lambda x: (x['value1']/x['value2']).sum()) })  Output:  value1 value2 value3 value4 id flag 1 A 385 16 290.416667 97.272727 2 B 270 9 82.500000 205.714286
Question: The following is an example of items rated by 1,2 or 3 stars. I am trying to count all combinations of item ratings (stars) per month. In the following example, item 10 was rated in month 1 and has two ratings equal 1, one rating equal 2 and one rating equal 3. inp = pd.DataFrame({'month':[1,1,1,1,1,2,2,2], 'item':[10,10,10,10,20,20,20,20], 'star':[1,2,1,3,3,2,2,3]} ) month item star 0 1 10 1 1 1 10 2 2 1 10 1 3 1 10 3 4 1 20 3 5 2 20 2 6 2 20 2 7 2 20 3  For the given above input frame output should be:  month item star_1_cnt star_2_cnt star_3_cnt 0 1 10 2 1 1 1 1 20 0 0 1 2 2 20 0 2 1  I am trying to solve the problem starting with the following code, which result still needs to be converted to the desired format of the output frame and which gives the wrong answers: 1 20 3 (1, 1) 2 20 3 (1, 1)  Anyway, there should be a better way to create the output table, then finalizing this one: months = [1,2] items = [10,20] stars = [1,2,3] d = {'month': [], 'item': [], 'star': [], 'star_cnts': [] } for month in months: for star in stars: for item in items: star_cnts=dict(inp[(inp['item']==item) & (inp['star']==star)].value_counts()).values() d['month'].append(month) d['item'].append(item) d['star'].append(star) d['star_cnts'].append(star_cnts) pd.DataFrame(d) month item star star_cnts 0 1 10 1 (2) 1 1 20 1 () 2 1 10 2 (1) 3 1 20 2 (2) 4 1 10 3 (1) 5 1 20 3 (1, 1) 6 2 10 1 (2) 7 2 20 1 () 8 2 10 2 (1) 9 2 20 2 (2) 10 2 10 3 (1) 11 2 20 3 (1, 1)  ​

Answer: This seems like a nice problem for pd.get_dummies: new_df = ( pd.concat([df, pd.get_dummies(df['star'])], axis=1) .groupby(['month', 'item'], as_index=False) [df['star'].unique()] .sum() )  Output: >>> new_df month item 1 2 3 0 1 10 2 1 1 1 1 20 0 0 1 2 2 20 0 2 1  Renaming, too: u = df['star'].unique() new_df = ( pd.concat([df, pd.get_dummies(df['star'])], axis=1) .groupby(['month', 'item'], as_index=False) [u] .sum() .rename({k: f'star_{k}_cnt' for k in df['star'].unique()}, axis=1) )  Output: >>> new_df month item star_1_cnt star_2_cnt star_3_cnt 0 1 10 2 1 1 1 1 20 0 0 1 2 2 20 0 2 1  Obligatory one- (or two-) liners: # Renames the columns u = df['star'].unique() new_df = pd.concat([df, pd.get_dummies(df['star'])], axis=1).groupby(['month', 'item'], as_index=False)[u].sum().rename({k: f'star_{k}_cnt' for k in df['star'].unique()}, axis=1)
Question: I have a dictionary of the form: {"level": [1, 2, 3], "conf": [-1, 1, 2], "text": ["here", "hel", "llo"]}  I want to filter the lists to remove every item at index i where an index in the value "conf" is not >0. So for the above dict, the output should be this: {"level": [2, 3], "conf": [1, 2], "text": ["hel", "llo"]}  As the first value of conf was not > 0. I have tried something like this: new_dict = {i: [a for a in j if a >= min_conf] for i, j in my_dict.items()}  But that would work just for one key.

Answer: I solved it with this: from typing import Dict, List, Any, Set d = {"level":[1,2,3], "conf":[-1,1,2], "text":["-1", "hel", "llo"]} # First, we create a set that stores the indices which should be kept. # I chose a set instead of a list because it has a O(1) lookup time. # We only want to keep the items on indices where the value in d["conf"] is greater than 0 filtered_indexes = {i for i, value in enumerate(d.get('conf', [])) if value > 0} def filter_dictionary(d: Dict[str, List[Any]], filtered_indexes: Set[int]) -> Dict[str, List[Any]]: filtered_dictionary = d.copy() # We'll return a modified copy of the original dictionary for key, list_values in d.items(): # In the next line the actual filtering for each key/value pair takes place. # The original lists get overwritten with the filtered lists. filtered_dictionary[key] = [value for i, value in enumerate(list_values) if i in filtered_indexes] return filtered_dictionary print(filter_dictionary(d, filtered_indexes))  Output: {'level': [2, 3], 'conf': [1, 2], 'text': ['hel', 'llo']}
Question: I need a unique (unsigned int) id for my python data class. This is very similar to this so post, but without explicit ctors. import attr from attrs import field from itertools import count @attr.s(auto_attribs=True) class Person: #: each Person has a unique id _counter: count[int] = field(init=False, default=count()) _unique_id: int = field(init=False) @_unique_id.default def _initialize_unique_id(self) -> int: return next(self._counter)  Is there any more-"pythonic" solution?

Answer: Use a default factory instead of just a default. This allows to define a call to get the next id on each instantiation. A simple means to get a callable that counts up is to use count().__next__, the equivalent of calling next(...) on a count instance.1 The common "no explicit ctor" libraries attr and dataclasses both support this: from itertools import count from dataclasses import dataclass, field @dataclass class C: identifier: int = field(default_factory=count().__next__) import attr @attr.s class C: identifier: int = attr.field(factory=count().__next__)  To always use the automatically generated value and prevent passing one in as a parameter, use init=False. @dataclass class C: identifier: int = field(default_factory=count().__next__, init=False)   1 If one wants to avoid explicitly addressing magic methods, one can use a closure over a count. For example, factory=lambda counter=count(): next(counter).
Question: I have 3 columns namely Models(should be taken as index), Accuracy without normalization, Accuracy with normalization (zscore, minmax, maxabs, robust) and these are required to be created as:  ------------------------------------------------------------------------------------ | Models | Accuracy without normalization | Accuracy with normalization | | | |-----------------------------------| | | | zscore | minmax | maxabs | robust | ------------------------------------------------------------------------------------  dfmod-> Models column dfacc-> Accuracy without normalization dfacc1-> Accuracy with normalization - zscore dfacc2-> Accuracy with normalization - minmax dfacc3-> Accuracy with normalization - maxabs dfacc4-> Accuracy with normalization - robust  dfout=pd.DataFrame({('Accuracy without Normalization'):{dfacc}, ('Accuracy using Normalization','zscore'):{dfacc1}, ('Accuracy using Normalization','minmax'):{dfacc2}, ('Accuracy using Normalization','maxabs'):{dfacc3}, ('Accuracy using Normalization','robust'):{dfacc4}, },index=dfmod )  I was trying to do something like this but i can't figure out any further Test data: qda 0.6333 0.6917 0.5917 0.6417 0.5833 svm 0.5333 0.6917 0.5333 0.575 0.575 lda 0.5333 0.6583 0.5333 0.5667 0.5667 lr 0.5333 0.65 0.4917 0.5667 0.5667 dt 0.5333 0.65 0.4917 0.5667 0.5667 rc 0.5083 0.6333 0.4917 0.525 0.525 nb 0.5 0.625 0.475 0.5 0.4833 rfc 0.5 0.625 0.4417 0.4917 0.4583 knn 0.3917 0.6 0.4417 0.4833 0.45 et 0.375 0.5333 0.4333 0.4667 0.45 dc 0.375 0.5333 0.4333 0.4667 0.425 qds 0.3417 0.5333 0.4 0.4583 0.3667 lgt 0.3417 0.525 0.3917 0.45 0.3583 lt 0.2333 0.45 0.3917 0.4167 0.3417  These are values for respective subcolumns in order specified in the table above

Answer: There's a dirty way to do this, I'll write about it till someone answers with a better idea. Here we go: import pandas as pd # I assume that you can read raw data named test.csv by pandas and # set header = None cause you mentioned the Test data without any headers, so: df = pd.read_csv("test.csv", header = None) # Then define preferred Columns! MyColumns = pd.MultiIndex.from_tuples([("Models" , ""), ("Accuracy without normalization" , ""), ("Accuracy with normalization" , "zscore"), ("Accuracy with normalization" , "minmax"), ("Accuracy with normalization" , "maxabs"), ("Accuracy with normalization" , "robust")]) # Create new DataFrame with specified Columns, after this you should pass values New_DataFrame = pd.DataFrame(df , columns = MyColumns) # a loop for passing values for item in range(len(MyColumns)): New_DataFrame.loc[: , MyColumns[item]] = df.iloc[: , item]  This gives me:  after all, if you want to set Models as the index of New_DataFrame, You can continue with: New_DataFrame.set_index(New_DataFrame.columns[0][0] , inplace=True) New_DataFrame  This gives me:
Question: I have this CLI tool called Rackfocus. I've published to PyPI, and I'm reasonably sure it worked just fine before. When I try to run it with current versions of Python on Mac, I get the error: No module named rackfocus.__main__; 'rackfocus' is a package and cannot be directly executed  All I want is one package with one entry point that users can download and use using pip. Based on tutorials, I have this in setup.py: packages=['rackfocus'] entry_points = { 'console_scripts': [ 'rackfocus=rackfocus.run:main' ] }  And I have a rackfocus.run:main function, an init.py and everything. What's wrong? You can reproduce this locally:  Clone my repo. Create and activate a virtualenv (optional). pip3 install -e . python3 -m rackfocus 

Answer: entry_points = { 'console_scripts': [ 'rackfocus=rackfocus.run:main' ] }  This tells the packaging system to create a wrapper executable named rackfocus. That executable will automatically handle all the necessary steps to get Python off the ground, find the run module in the rackfocus package, find its main function and call it. You run the executable like rackfocus (if you are using a virtual environment, it should be on the path already), not python -m rackfocus. Using python -m rackfocus is completely unrelated to that (it doesn't even have anything to do with packaging, and can easily be used with code that hasn't been installed yet). It doesn't use the wrapper; instead, it simply attempts to execute the rackfocus module. But in your case, rackfocus isn't a module; it's a package. The error message means exactly what it says. You would want python -m rackfocus.run to execute the run module - but of course, that still doesn't actually call main() (just like it wouldn't with python rackfocus/main.py - though the -m approach is more powerful; in particular, it allows your relative imports to work). The error message says rackfocus.__main__ because you can make a package runnable by giving it a __main__ module.
Question: For the below code # A simple generator function def infinite_sequence(): num = 1 while True: yield num num += 1 aaa = infinite_sequence() bbb = infinite_sequence() ccc = infinite_sequence() print(next(aaa)) print(next(aaa)) print(next(bbb)) print(next(bbb)) print(next(ccc)) print(next(ccc))  the output is: 1 2 1 2 1 2 When trying to call the same generator functions from another function, the output is not as expected def switchAction(input): action_item = { "A": next(aaa), "B": next(bbb), "C": next(ccc) } return action_item.get(input) print(switchAction("A")) print(switchAction("A")) print(switchAction("B")) print(switchAction("B")) print(switchAction("C")) print(switchAction("C"))  The output is: 1 2 3 4 5 6 Why the counter is continuing across generator functions in the case of calling from another function? How to achieve the same output as that of first case for the second case above.

Answer: The problem is you call next on all values every time you call switchAction, since you define the dict over and over again. A solution to your problem can be as follows: # A simple generator function def infinite_sequence(): num = 1 while True: yield num num += 1 aaa = infinite_sequence() bbb = infinite_sequence() ccc = infinite_sequence() action_item = { "A": aaa, "B": bbb, "C": ccc } # create the dict one time, and don't call next within the definition. def switchAction(name): # also input is a dangerous var name so change it to name return next(action_item.get(name)) print(switchAction("A")) print(switchAction("A")) print(switchAction("B")) print(switchAction("B")) print(switchAction("C")) print(switchAction("C"))
Question: I have two lists. I want to create a Literal using both these lists category1 = ["image/jpeg", "image/png"] category2 = ["application/pdf"] SUPPORTED_TYPES = typing.Literal[category1 + category2]  Is there any way to do this? I have seen the question typing: Dynamically Create Literal Alias from List of Valid Values but this doesnt work for my use case because I dont want mimetype to be of type typing.Tuple. I will be using the Literal in a function - def process_file(filename: str, mimetype: SUPPORTED_TYPES)  What I have tried - supported_types_list = category1 + category2 SUPPORTED_TYPES = Literal[supported_types_list] SUPPORTED_TYPES = Literal[*supported_types_list] # this gives 2 different literals, rather i want only 1 literal SUPPORTED_TYPES = Union[Literal["image/jpeg", "image/png"], Literal["application/pdf"]] 

Answer: Use the same technique as in the question you linked: build the lists from the literal types, instead of the other way around: SUPPORTED_IMAGE_TYPES = typing.Literal["image/jpeg", "image/png"] SUPPORTED_OTHER_TYPES = typing.Literal["application/pdf"] SUPPORTED_TYPES = typing.Literal[SUPPORTED_IMAGE_TYPES, SUPPORTED_OTHER_TYPES] category1 = list(typing.get_args(SUPPORTED_IMAGE_TYPES)) category2 = list(typing.get_args(SUPPORTED_OTHER_TYPES))  The only part of this that wasn't already covered in the other answer is SUPPORTED_TYPES = typing.Literal[SUPPORTED_IMAGE_TYPES, SUPPORTED_OTHER_TYPES], which, yeah, you can do that. It's equivalent to your original definition of SUPPORTED_TYPES.
Question: Assume we have: @pytest.fixture() def setup(): print('All set up!') return True def foo(setup): print('I am using a fixture to set things up') setup_done=setup  I'm looking for a way to get to know caller function name (in this case: foo) from within setup fixture. So far I have tried: import inspect @pytest.fixture() def setup(): daddy_function_name = inspect.stack()[1][3] print(daddy_function_name) print('All set up!') return True  But what gets printed is: call_fixture_func How do I get foo from printing daddy_function_name?

Answer: You can use the built-in request fixture in your own fixture:  The request fixture is a special fixture providing information of the requesting test function.  Its node attribute is the  Underlying collection node (depends on current request scope).  import pytest @pytest.fixture() def setup(request): return request.node.name def test_foo(setup): assert setup == "test_foo"