eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm attempting to create a function to load Sagemaker models within a jupyter notebook using shell commands. The problem arises when I try to store the function in a utilities.py file and source it for multiple notebooks. Here are the contents of the utilities.py file that I am sourcing in a jupyter lab notebook. def get_aws_sagemaker_model(model_loc): """ TO BE USED IN A JUPYTER NOTEBOOK extracts a sagemaker model that has ran and been completed deletes the copied items and leaves you with the model note that you will need to have the package installed with correct versioning for whatever model you have trained ie. if you are loading an XGBoost model, have XGBoost installed Args: model_loc (str) : s3 location of the model including file name Return: model: unpacked and loaded model """ import re import tarfile import os import pickle as pkl # extract the filename from beyond the last backslash packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2] # copy and paste model file locally command_string = "!aws s3 cp {model_loc} ." exec(command_string) # use tarfile to extract tar = tarfile.open(packed_model_name) # extract filename from tarfile unpacked_model_name = tar.getnames()[0] tar.extractall() tar.close() model = pkl.load(open(unpacked_model_name, 'rb')) # cleanup copied files and unpacked model os.remove(packed_model_name) os.remove(unpacked_model_name) return model  The error occurs when trying to execute the command string: Traceback (most recent call last): File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_10889/996524724.py", line 1, in <module> model = get_aws_sagemaker_model("my-model-loc") File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model exec(command_string) File "<string>", line 1 !aws s3 cp my-model-loc . ^ SyntaxError: invalid syntax  It seems like jupyter isn't receiving the command before exec checks the syntax. Is there a way around this besides copying the function into each jupyter notebook that I use? Thank you!

Answer: You can use transform_cell method of IPython's shell to transform the IPython syntax into valid plain-Python: from IPython import get_ipython ipython = get_ipython() code = ipython.transform_cell('!ls') print(code)  which will show:  get_ipython().system('!ls')  You can use that as input for exec: exec(code)  Or directly: exec(ipython.transform_cell('!ls'))
Question: I have a config tree such as: config.yaml model/ model_a.yaml model_b.yaml model_c.yaml  Where config.yaml contains: # @package _global_ defaults: - _self_ - model: model_a.yaml some_var: 42  I would like to access the name of the model config file used (either the default or overridden) from my python code or from the file itself. Something like: @hydra.main(...) def main(config): model_name = config.model.__filename__  or (from e.g. model_a.yaml) dropout: true dense_layers: 128 model_name: ${__filename__}  Thanks in advance!

Answer: The a look at the hydra.runtime.choices variable mentioned in the Configuring Hydra - Introduction page of the Hydra docs. This variable stores a mapping that describes each of the choices that Hydra has made in composing the output config. Using your example from above with model: model_a.yaml in the defaults list: # my_app.py import hydra from pprint import pprint from hydra.core.hydra_config import HydraConfig from omegaconf import OmegaConf @hydra.main(config_path=".", config_name="config") def main(config): hydra_cfg = HydraConfig.get() print("choice of model:") pprint(OmegaConf.to_container(hydra_cfg.runtime.choices)) main()  At the command line: $ python3 app.py choices used: {'hydra/callbacks': None, 'hydra/env': 'default', 'hydra/help': 'default', 'hydra/hydra_help': 'default', 'hydra/hydra_logging': 'default', 'hydra/job_logging': 'default', 'hydra/launcher': 'basic', 'hydra/output': 'default', 'hydra/sweeper': 'basic', 'model': 'model_a.yaml'}  As you can see, in this example the config option model_a.yaml is stored in the Hydra config at hydra_cfg.runtime.choices.model.
Question: Here is the little project of Cancer detection, and it has already has the dataset and colab code, but I get an error when I execute model.fit(x_train, y_train, epochs=1000)  The error is: ValueError: Input 0 of layer "sequential" is incompatible with the layer: expected shape=(None, 455, 30), found shape=(None, 30)  When I look at the comments, other people are having this issue

Answer: The Tensorflow model expects the first dimension of the input to be the batch size, in the model declaration however they set the input shape to be the same shape as the input. To fix this you can change the input shape of the model to be the number of feature in the dataset. model.add(tf.keras.layers.Dense(256, input_shape=(x_train.shape[1],), activation='sigmoid'))  The number of rows in the .csv files will be the number of samples in your dataset. Since you're not using batches, the model will evaluate the whole dataset at once every epoch
Question: I have a Python3.9 / Quart / Hypercorn microservice which runs in a conda environment configured with an environment.yml file. The base image is continuumio/miniconda3. It took a lot of hacks to get this launching because of conda init issues etc. Is there a cleaner way to get a conda environment installed and activated within Docker without having to resort to conda run commands and override the default SHELL commands? FROM continuumio/miniconda3 COPY . /api/ WORKDIR /api/src # See this tutorial for details https://pythonspeed.com/articles/activate-conda-dockerfile/ RUN conda env create -f /api/conda_environment_production.yml SHELL ["conda", "run", "-n", "ms-amazing-environment", "/bin/bash", "-c"] ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "ms-amazing-environment", "hypercorn", "--bind", "0.0.0.0:5000", "QuartAPI:app"] EXPOSE 5000 

Answer: An alternative approach is described here. Basically you can activate conda environment within bash script and run you commands there. entrypoint.sh: #!/bin/bash --login # The --login ensures the bash configuration is loaded, # Temporarily disable strict mode and activate conda: set +euo pipefail conda activate ms-amazing-environment # enable strict mode: set -euo pipefail # exec the final command: exec hypercorn --bind 0.0.0.0:5000 QuartAPI:app  Dockerfile: FROM continuumio/miniconda3 COPY . /api/ WORKDIR /api/src # See this tutorial for details https://pythonspeed.com/articles/activate-conda-dockerfile/ RUN conda env create -f /api/conda_environment_production.yml # The code to run when container is started: COPY entrypoint.sh ./ ENTRYPOINT ["./entrypoint.sh"] EXPOSE 5000
Question: I'm not able to authenticate using my twitter developer account even though my account active import tweepy consumer_key= 'XX1' consumer_secret= 'XX2' access_token= 'XX3' access_token_secret= 'XX4' auth = tweepy.OAuthHandler(consumer_key, consumer_secret) auth.set_access_token(access_token, access_token_secret) api = tweepy.API(auth) api.update_status("Hello Tweepy")  i'm getting error : Forbidden: 403 Forbidden 453 - You currently have Essential access which includes access to Twitter API v2 endpoints only. If you need access to this endpoint, you’ll need to apply for Elevated access via the Developer Portal. You can learn more here: https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve  There is no option to move to Essential to Elevated on the developer portal. any suggestion ?

Answer: I found out that Essential access can use only Twitter API v2 The code should be import tweepy consumer_key= 'x' consumer_secret= 'xx' access_token= 'xxx' access_token_secret= 'xxxx' client = tweepy.Client(consumer_key= consumer_key,consumer_secret= consumer_secret,access_token= access_token,access_token_secret= access_token_secret) query = 'news' tweets = client.search_recent_tweets(query=query, max_results=10) for tweet in tweets.data: print(tweet.text)  Thanks to https://twittercommunity.com/t/403-forbidden-using-tweepy/162435/2
Question: I just downloaded pytube (version 11.0.1) and started with this code snippet from here: from pytube import YouTube YouTube('https://youtu.be/9bZkp7q19f0').streams.first().download()  which gives this error: AttributeError Traceback (most recent call last) <ipython-input-29-0bfa08b87614> in <module> ----> 1 YouTube('https://youtu.be/9bZkp7q19f0').streams.first().download() ~/anaconda3/lib/python3.8/site-packages/pytube/__main__.py in streams(self) 290 """ 291 self.check_availability() --> 292 return StreamQuery(self.fmt_streams) 293 294 @property ~/anaconda3/lib/python3.8/site-packages/pytube/__main__.py in fmt_streams(self) 175 # https://github.com/pytube/pytube/issues/1054 176 try: --> 177 extract.apply_signature(stream_manifest, self.vid_info, self.js) 178 except exceptions.ExtractError: 179 # To force an update to the js file, we clear the cache and retry ~/anaconda3/lib/python3.8/site-packages/pytube/extract.py in apply_signature(stream_manifest, vid_info, js) 407 408 """ --> 409 cipher = Cipher(js=js) 410 411 for i, stream in enumerate(stream_manifest): ~/anaconda3/lib/python3.8/site-packages/pytube/cipher.py in __init__(self, js) 42 43 self.throttling_plan = get_throttling_plan(js) ---> 44 self.throttling_array = get_throttling_function_array(js) 45 46 self.calculated_n = None ~/anaconda3/lib/python3.8/site-packages/pytube/cipher.py in get_throttling_function_array(js) 321 322 array_raw = find_object_from_startpoint(raw_code, match.span()[1] - 1) --> 323 str_array = throttling_array_split(array_raw) 324 325 converted_array = [] ~/anaconda3/lib/python3.8/site-packages/pytube/parser.py in throttling_array_split(js_array) 156 # Handle functions separately. These can contain commas 157 match = func_regex.search(curr_substring) --> 158 match_start, match_end = match.span() 159 160 function_text = find_object_from_startpoint(curr_substring, match.span()[1]) AttributeError: 'NoneType' object has no attribute 'span'  and I wonder why? Can anyone help me? I am running this snippet in an ipython console (IPython version 7.22.0) with Python 3.8.8 in a conda environment.

Answer: Found this issue, pytube v11.0.1. It's a little late for me, but if no one has submitted a fix tomorrow I'll check it out. in C:\Python38\lib\site-packages\pytube\parser.py Change this line: 152: func_regex = re.compile(r"function\([^)]+\)") to this: 152: func_regex = re.compile(r"function\([^)]?\)") The issue is that the regex expects a function with an argument, but I guess youtube added some src that includes non-paramterized functions.
Question: I have the following context manager and decorator to time any given function or code block: import time from contextlib import ContextDecorator class timer(ContextDecorator): def __init__(self, label: str): self.label = label def __enter__(self): self.start_time = time.perf_counter() return self def __exit__(self, *exc): net_time = time.perf_counter() - self.start_time print(f"{self.label} took {net_time:.1f} seconds") return False  You can use it as a context manager: with timer("my code block"): time.sleep(2) # my code block took 2.0 seconds  You can also use it as a decorator: @timer("my_func") def my_func(): time.sleep(3) my_func() # my_func took 3.0 seconds  The only thing I don't like is having to manually pass the function name as the label when it's used as a decorator. I would love for the decorator to automatically use the function name if no label is passed: @timer() def my_func(): time.sleep(3) my_func() # my_func took 3.0 seconds  Is there any way to do this?

Answer: If you also override the __call__() method inherited from the ContextDecorator base class in your timer class, and add a unique default value to the initializer for the label argument, you can check for that and grab the function's __name__ when it's called: import time from contextlib import ContextDecorator class timer(ContextDecorator): def __init__(self, label: str=None): self.label = label def __call__(self, func): if self.label is None: # Label was not provided self.label = func.__name__ # Use function's name. return super().__call__(func) def __enter__(self): self.start_time = time.perf_counter() return self def __exit__(self, *exc): net_time = time.perf_counter() - self.start_time print(f"{self.label} took {net_time:.1f} seconds") return False @timer() def my_func(): time.sleep(3) my_func() # -> my_func took 3.0 seconds
Question: pip freeze output: aiohttp==3.8.1 aiosignal==1.2.0 alembic==1.7.5 aniso8601==9.0.1 async-timeout==4.0.1 attrs==21.2.0 base58==2.1.1 bitarray==1.2.2 certifi==2021.10.8 charset-normalizer==2.0.7 click==8.0.3 cytoolz==0.11.2 eth-abi==2.1.1 eth-account==0.5.6 eth-hash==0.3.2 eth-keyfile==0.5.1 eth-keys==0.3.3 eth-rlp==0.2.1 eth-typing==2.2.2 eth-utils==1.10.0 Flask==2.0.2 flask-marshmallow==0.14.0 Flask-Migrate==3.1.0 Flask-RESTful==0.3.9 Flask-Script==2.0.6 Flask-SQLAlchemy==2.5.1 frozenlist==1.2.0 hexbytes==0.2.2 idna==3.3 ipfshttpclient==0.8.0a2 itsdangerous==2.0.1 Jinja2==3.0.3 jsonschema==3.2.0 lru-dict==1.1.7 Mako==1.1.6 MarkupSafe==2.0.1 marshmallow==3.14.1 marshmallow-sqlalchemy==0.26.1 multiaddr==0.0.9 multidict==5.2.0 netaddr==0.8.0 parsimonious==0.8.1 protobuf==3.19.1 psycopg2==2.9.2 pycryptodome==3.11.0 pyrsistent==0.18.0 pytz==2021.3 requests==2.26.0 rlp==2.0.1 six==1.16.0 SQLAlchemy==1.4.27 toolz==0.11.2 typing_extensions==4.0.0 urllib3==1.26.7 varint==1.0.2 web3==5.25.0 websockets==9.1 Werkzeug==2.0.2 yarl==1.7.2  Python version: 3.10.0 I installed Web3 using the pip install web3 command in my venv. To create my venv, I did virtualenv -p python3 venv, so I don't think there is an issue with the virtual env. However in my test.py when I do the following: from web3 import Web3  I get a traceback error that there is no module named "web3"

Answer: Are you sourcing your venv before running test.py? If so, then try this, source venv/bin/activate pip uninstall web3==5.25.0 pip install web3==5.25.0 python test.py  (Since your pip freeze is correct), try this as well which python  This should give you the python bin that is currently being used by your shell. (Check if the path you get is the venv one).
Question: I have to write a function which has 2 nested functions inside and computes only sum or difference of as many numbers as we want. Every equation end with "=". I wrote sth like this but it still doesn't work. What I am doing wrong? def calculator(x: int): def operation(operator: str): def calculation(y: int): while operator != "=": if operators[i] == "=": result -= digits[j] elif operators[i] == "+": result += digits[j] else: break return result return operator return calculator(result)  Function should work like that: calculator(1)('+')(4)('-')(2)('=')  The result is 3. I can't use any package import or global variables.

Answer: You can also use a dictionary to define your operations, and make a very readable and simple function with space for expansion: def calc(a: int): return lambda op: { '+': lambda b: calc(a+b), '-': lambda b: calc(a-b), '/': lambda b: calc(a/b), '*': lambda b: calc(a*b), }.get(op, a) print(calc(2)('+')(3)('-')(10)('/')(10)('*')(-100)('='))
Question: While trying to migrate pyqt5 code to pyqt6, i have occured a problem with setWindowFlags: self.setWindowFlags(Qt.WindowStaysOnTopHint) returns an error: AttributeError: type object 'Qt' has no attribute 'WindowStaysOnTopHint'. So wat is the similar in PyQt6?

Answer: QtCore.Qt.WindowType.WindowStaysOnTopHint
Question: Consider the below dataframe: import pandas as pd from numpy import nan data = [ (111, nan, nan, 111), (112, 112, nan, 115), (113, nan, nan, nan), (nan, nan, nan, nan), (118, 110, 117, nan), ] df = pd.DataFrame(data, columns=[f'num{i}' for i in range(len(data[0]))]) num0 num1 num2 num3 0 111.0 NaN NaN 111.0 1 112.0 112.0 NaN 115.0 2 113.0 NaN NaN NaN 3 NaN NaN NaN NaN 4 118.0 110.0 117.0 NaN  Assuming my index is unique, I'm looking to retrieve the unique values per index row, to an output like the one below. I wish to keep the empty rows.  num1 num2 num3 0 111.0 NaN NaN 1 112.0 115.0 NaN 2 113.0 NaN NaN 3 NaN NaN NaN 4 110.0 117.0 118.0  I have a working, albeit slow, solution, see below. The output number order is not relevant, as long all values are presented to the leftmost column and nulls to the right. I'm looking for best practices and potential ideas to speed up the code. Thank you in advance. def arrange_row(row): values = list(set(row.dropna(axis=1).values[0])) values = [nan] if not values else values series = pd.Series(values, index=[f"num{i}" for i in range(1, len(values)+1)]) return series df.groupby(level=-1).apply(arrange_row).unstack(level=-1) pd.version == '1.2.3' 

Answer: Another option, albeit longer: outcome = (df.melt(ignore_index= False) # keep the index as a tracker .reset_index() # get the unique rows .drop_duplicates(subset=['index','value']) .dropna() # use this to build the new column names .assign(counter = lambda df: df.groupby('index').cumcount() + 1) .pivot('index', 'counter', 'value') .add_prefix('num') .reindex(df.index) .rename_axis(columns=None) ) outcome num1 num2 num3 0 111.0 NaN NaN 1 112.0 115.0 NaN 2 113.0 NaN NaN 3 NaN NaN NaN 4 118.0 110.0 117.0  If you want it to exactly match your output, you can dump it into numpy, sort and return to pandas: pd.DataFrame(np.sort(outcome, axis = 1), columns = outcome.columns) num1 num2 num3 0 111.0 NaN NaN 1 112.0 115.0 NaN 2 113.0 NaN NaN 3 NaN NaN NaN 4 110.0 117.0 118.0  Another option is to do the sorting within numpy before reshaping in Pandas: (pd.DataFrame(np.sort(df, axis = 1)) .apply(pd.unique, axis=1) .apply(pd.Series) .dropna(how='all',axis=1) .set_axis(['num1', 'num2','num3'], axis=1) ) num1 num2 num3 0 111.0 NaN NaN 1 112.0 115.0 NaN 2 113.0 NaN NaN 3 NaN NaN NaN 4 110.0 117.0 118.0
Question: I am using nsidnev/fastapi-realworld-example-app. I need to apply transaction logic to this project. In one API, I am calling a lot of methods from repositories and doing updating, inserting and deleting operations in many tables. If there is an exception in any of these operations, how can I roll back changes? (Or if everything is correct then commit.)

Answer: nsidnev/fastapi-realworld-example-app is using asyncpg. There are two ways to use Transactions. 1. async with statement async with conn.transaction(): await repo_one.update_one(...) await repo_two.insert_two(...) await repo_three.delete_three(...) # This automatically rolls back the transaction: raise Exception  2. start, rollback, commit statements tx = conn.transaction() await tx.start() try: await repo_one.update_one(...) await repo_two.insert_two(...) await repo_three.delete_three(...) except: await tx.rollback() raise else: await tx.commit()   Getting the connection conn in routes Inject conn: Connection = Depends(_get_connection_from_pool). from asyncpg.connection import Connection from fastapi import Depends from app.api.dependencies.database import _get_connection_from_pool @router.post( ... ) async def create_new_article( ... conn: Connection = Depends(_get_connection_from_pool), # Add this ) -> ArticleInResponse:
Question: I have a dataframe: id group x1 A x1 B x2 A x2 A x3 B  I would like to create a new column new_group with the following conditions: If there are 2 unique group values within in the same id such as group A and B from rows 1 and 2, new_group should have "two" as its value. If there are only 1 unique group values within the same id such as group A from rows 3 and 4, the value for new_group should be that same group A. Otherwise, specify B. This is what I am looking for: id group new_group x1 A two x1 B two x2 A A x2 A A x3 B B  I tried something like this but don't know how to capture all the if-else conditions df.groupby("id")["group"].filter(lambda x: x.nunique() == 2)

Answer: Almost there. Change filter to transform and use a condition: df['new_group'] = df.groupby("id")["group"] \ .transform(lambda x: 'two' if (x.nunique() == 2) else x) print(df) # Output: id group new_group 0 x1 A two 1 x1 B two 2 x2 A A 3 x2 A A 4 x3 B B
Question: I have a dataframe like below. How can I get it into a nested dictionary like  Guest GuestCode ProductName Quantity Invoice No 0 Maria NaN Pro Plus Cream 2 OBFL22511 1 Maria NaN Soothe Stress Cream 1 OBFL22511 2 Sanchez OBFLG3108 Pro Plus Cream 1 OBFL22524 3 Karen OBFLG1600 Soothe Stress Cream 1 OBFL22525 4 Karen OBFLG1600 Pro Plus Cream 1 OBFL22525  I want the dataframe converted into the following dictionary format: {"Guest": {"GuestCode": {"Invoice No": {"ProductName": Quantity}}}  For example: {"Karen": {"OBFLG160": {"OBFL22525": {"Soothe Stress Cream": 1, "Pro Plus Cream": 1}}}  I tried this: for index, row in df.iterrows(): my_dict[row['Guest']] = {row['GuestCode']: {row['Invoice No']: {row['ProductName']}}}  But it does not list all the items if a guest has multiple products. I also tried and played around with this, but don't really understand dictionary comprehension: d = {k: v.groupby('GuestCode')['Invoice No','ProductName' , 'Quantity'].apply(list).to_dict() for k, v in df.groupby('Guest')} 

Answer: my_dict = {k[0]: {k[1]: {k[2]: {p: q for p, q in row[['ProductName', 'Quantity']].values}}} for k, row in df.fillna('<NA>').groupby(['Guest', 'GuestCode', 'Invoice No'])}  Output: >>> my_dict {'Karen': {'OBFLG1600': {'OBFL22525': {'Soothe Stress Cream': 1, 'Pro Plus Cream': 1}}}, 'Maria': {'<NA>': {'OBFL22511': {'Pro Plus Cream': 2, 'Soothe Stress Cream': 1}}}, 'Sanchez': {'OBFLG3108': {'OBFL22524': {'Pro Plus Cream': 1}}}} >>> import json >>> print(json.dumps(my_dict, indent=2)) { "Karen": { "OBFLG1600": { "OBFL22525": { "Soothe Stress Cream": 1, "Pro Plus Cream": 1 } } }, "Maria": { "<NA>": { "OBFL22511": { "Pro Plus Cream": 2, "Soothe Stress Cream": 1 } } }, "Sanchez": { "OBFLG3108": { "OBFL22524": { "Pro Plus Cream": 1 } } } }
Question: I am very new to using the Twitter API and was testing some Python code (below) from tweepy import OAuthHandler from tweepy import Stream import twitter_credentials class StdOutListener(Stream): def on_data(self, data): print(data) return True def on_error(self, status): print(status) twitter_stream = StdOutListener( twitter_credentials.CONSUMER_KEY, twitter_credentials.CONSUMER_KEY_SECERET, twitter_credentials.ACCESS_TOKEN, twitter_credentials.ACCESS_TOKEN_SECERET ) twitter_stream.filter(track=['#bitcoin'])  but whenever I try to run it, it would give this repeating error "Stream encountered HTTP error: 403". I checked the Twitter API response codes and error 403 was listed as the request is forbidden. Here are some troubleshooting steps I already took:  tried new access keys/consumer keys tried to create a new developer account uninstall and reinstall tweepy  None of these worked for me. So what is causing this error and how can I fix it? Thank you.

Answer: If you have Essential access, you won’t be able to access Twitter API v1.1. See the FAQ section about this in Tweepy's documentation for more information.