eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: import matplotlib.pyplot as plt  output ImportError Traceback (most recent call last) <ipython-input-7-a0d2faabd9e9> in <module> ----> 1 import matplotlib.pyplot as plt ~\AppData\Roaming\Python\Python38\site-packages\matplotlib\__init__.py in <module> 172 173 --> 174 _check_versions() 175 176 ~\AppData\Roaming\Python\Python38\site-packages\matplotlib\__init__.py in _check_versions() 157 # Quickfix to ensure Microsoft Visual C++ redistributable 158 # DLLs are loaded before importing kiwisolver --> 159 from . import ft2font 160 161 for modname, minver in [ ImportError: cannot import name 'ft2font' from partially initialized module 'matplotlib' (most likely due to a circular import) (C:\Users\p****\AppData\Roaming\Python\Python38\site-packages\matplotlib\__init__.py)  ​

Answer: As you are on a windows machine, there is a possible duplicate. Navigate by clicking here. This could be an issue regarding matplotlib. A force reinstall over pip would solve the issue. pip install matplotlib --force-reinstall  If you are working on Anaconda, launch Anaconda as Administrator, conda install freetype --force-reinstall  This solved the same issue for me.
Question: I don't understand how R handles the Python environment and Python version and keep getting the error Error: could not find a Python environment for /usr/bin/python. I installed Miniconda and created a conda environment in the shell: conda activate r-reticulate  Then, in R, I try to install keras (same problem with package tensorflow): library(keras) reticulate::use_condaenv() install_keras(method = "conda", conda = reticulate::conda_binary())  ... and get the following error: Error: could not find a Python environment for /usr/bin/python  I tried to figure out what Python R should be using by reticulate::py_config()  and get python: /usr/bin/python libpython: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib pythonhome: /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7 version: 2.7.16 (default, Jul 5 2020, 02:24:03) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.21) (-macos10.15-objc- numpy: /Users/bestocke/Library/Python/2.7/lib/python/site-packages/numpy numpy_version: 1.16.6 tensorflow: [NOT FOUND] python versions found: /usr/bin/python3 /usr/local/bin/python3 /usr/bin/python  I don't understand this. This seems to be using Python 2.7. When trying to figure out which Python is being used in the shell, I get: > which python /opt/miniconda3/envs/r-reticulate/bin/python  and > ls -l /opt/miniconda3/envs/r-reticulate/bin/python lrwxr-xr-x 1 username wheel 9 Aug 2 15:21 /opt/miniconda3/envs/r-reticulate/bin/python -> python3.6  Suggesting Python 3.6 should be used. What am I getting wrong here?

Answer: Try to follow the guide at https://tensorflow.rstudio.com/installation/: In your R-studio console :  install.packages("tensorflow")  library(tensorflow)  install_tensorflow()   If you have not installed Anaconda / Miniconda manually, then at step no. 3, a prompt will ask your permission to install Miniconda. If you already have conda installed, then :  Create new environment r-reticulate in conda : conda create -n r-reticulate Install tensorflow from R-studio console with parameters : install_tensorflow(method = 'conda', envname = 'r-reticulate') Load the reticulate package library(reticulate) Activate the conda environment in R-studio use_condaenv('r-reticulate') Load the tensorflow libray library(tensorflow) Check if tensorflow is active tf$constant("Hellow Tensorflow")  References :  https://tensorflow.rstudio.com/installation/ https://rstudio.github.io/reticulate/
Question: I am using keras with a tensorflow (version 2.2.0) backend to train a classifier to distinguish between two datasets, A and B, which I have mixed into a pandas DataFrame object x_train (with two columns), and with labels in a numpy array y_train. I would like to perform sample weighting in order to account for the fact that A has far more samples than B. In addition, A is comprised of two datasets A1 and A2, with A1 much larger than A2; I would like to account for this fact as well using my sample weights. I have the sample weights in a numpy array called w_train. There are ~10 million training samples. Here is example code: model = Sequential() model.add(Dense(64, input_dim=x_train.shape[1], activation='relu')) model.add(Dropout(0.1)) model.add(Dense(64, activation='relu')) model.add(Dropout(0.1)) model.add(Dense(64, activation='relu')) model.add(Dropout(0.1)) model.add(Dense(64, activation='relu')) model.add(Dropout(0.1)) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(x_train.iloc, y_train, sample_weight=w_train)  When I use the sample_weight argument in model.fit(), I find that the model fitting initialization (i.e. whatever happens before keras starts to display the training progress) takes forever, too long to wait for. The problem goes away when I limit the dataset to 1000 samples, but as I increase to 100000 or 1000000 samples I notice that there is a significant difference in initialization and fitting time, so I suspect it has something to do with the way the data is being loaded. Nevertheless, it seems weird that merely adding the sample_weights argument would cause such a large timing difference. Other information: I am running on CPU using a Jupyter notebook. What is the problem here? Is there a way for me to modify the training setup or something else in order to speed up the initialization (or training) time?

Answer: The issue is caused by how TensorFlow validates some type of input objects. Such validations, when the data are surely correct, are exclusively a wasted time expenditure (I hope in the future it will be handled better). In order to force TensorFlow to skip such validation procedures, you can trivially wrap the weights in a Pandas Series, such as follows: model.fit(x_train.iloc, y_train, sample_weight=pd.Series(w_train))  Do note that in your code you are using the metrics keyword. If you want the accuracy to be actually weighted on the provided weights, to use the weighted_metrics argument instead.
Question: I am learning the Pydantic module, trying to adopt its features/benefits via a toy FastAPI web backend as an example implementation. I chose to use Pydantic's SecretStr to "hide" passwords. I know it is not really secure, and I am also using passlib for proper password encryption in DB storage (and using HTTPS for security in transit). But this got me thinking: if there is no real security to SecretStr, what is its purpose? I don't mean for this to sound inflammatory; Pydantic does not claim that the Secret Types are secure. The only claim they provide is this:  You can use the SecretStr and the SecretBytes data types for storing sensitive information that you do not want to be visible in logging or tracebacks.  But I do not understand this: how does SecretStr help in hiding logging or tracebacks? Can't I just make sure not to log the password at all?  Can someone provide an explanation + example to help me better understand when and how it can be helpful? I am struggling to find its real purpose... and if there is no benefit, then it is better to just use an str for the model/schema instead of SecretStr.

Answer: You already answered a big part of the question yourself.  You can use the SecretStr and the SecretBytes data types for storing sensitive information that you do not want to be visible in logging or tracebacks.  I would like to add another benefit. Developers are constantly reminded that they are working with secrets because they need to invoke .get_secret_value() to read the real value. We might consider this as syntactic salt (analog to syntactic sugar which makes things easier). Syntactic salt makes things harder to mess up. If you would try to send a SecretStr in a response model in e.g. FastAPI, you'd need to proactively enable that functionality. From the docs: class SimpleModelDumpable(BaseModel): password: SecretStr class Config: json_encoders = { SecretStr: lambda v: v.get_secret_value() if v else None }
Question: I'm having trouble installing OpenCV with Conda. I tried running numerous commands, none of which worked. For example, when I ran conda install -c anaconda opencv (as per https://anaconda.org/anaconda/opencv) I get this error: UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation in your environment: Specifications: - opencv -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0'] Your python: python=3.8 If python is on the left-most side of the chain, that's the version you've asked for. When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that.  Why is this happening and how can I install OpenCV in Spyder? Thanks.

Answer: The answer for this issue is updating Anaconda to the newest version: conda update -n base -c defaults conda  then you can install opencv normally using: conda install -c conda-forge opencv
Question: I want to add additional Dense layer after pretrained TFDistilBertModel, TFXLNetModel and TFRobertaModel Huggingface models. I have already seen how I can do this with the TFBertModel, e.g. in this notebook: output = bert_model([input_ids,attention_masks]) output = output[1] output = tf.keras.layers.Dense(32,activation='relu')(output)  So, here I need to use the second item(i.e. item with index 1) of the BERT output tuple. According to the docs TFBertModel has pooler_output at this tuple index. But the other three models don't have pooler_output. So, how can I add additional layers to the other three model outputs?

Answer: It looks like pooler_output is a Roberta and Bert specific output. But instead of using pooler_output we can use a few hidden_states(so, not only last hidden state) with all models, we want to use them because papers report that hidden_states can give more accuracy than just one last_hidden_state. # Import the needed model(Bert, Roberta or DistilBert) with output_hidden_states=True transformer_model = TFBertForSequenceClassification.from_pretrained('bert-large-cased', output_hidden_states=True) input_ids = tf.keras.Input(shape=(128, ),dtype='int32') attention_mask = tf.keras.Input(shape=(128, ), dtype='int32') transformer = transformer_model([input_ids, attention_mask]) hidden_states = transformer[1] # get output_hidden_states hidden_states_size = 4 # count of the last states hiddes_states_ind = list(range(-hidden_states_size, 0, 1)) selected_hiddes_states = tf.keras.layers.concatenate(tuple([hidden_states[i] for i in hiddes_states_ind])) # Now we can use selected_hiddes_states as we want output = tf.keras.layers.Dense(128, activation='relu')(selected_hiddes_states) output = tf.keras.layers.Dense(1, activation='sigmoid')(output) model = tf.keras.models.Model(inputs = [input_ids, attention_mask], outputs = output) model.compile(tf.keras.optimizers.Adam(lr=1e-4), loss='binary_crossentropy', metrics=['accuracy'])
Question: I have:  context = torch.tensor(context, dtype=torch.long, device=self.device) context = context.unsqueeze(0) generated = context with torch.no_grad(): past_outputs = None for i in trange(num_words): print(i, num_words) inputs = {"input_ids": generated} outputs, past_outputs = self.model( **inputs, past=past_outputs ) next_token_logits = outputs[ 0, -1, :] / (temperature if temperature > 0 else 1.0) # reptition penalty from CTRL # (https://arxiv.org/abs/1909.05858) for _ in set(generated.view(-1).tolist()): next_token_logits[_] /= repetition_penalty filtered_logits = top_k_top_p_filtering( next_token_logits, top_k=top_k, top_p=top_p) if temperature == 0: # greedy sampling: next_token = torch.argmax(filtered_logits).unsqueeze(0) else: next_token = torch.multinomial( F.softmax(filtered_logits, dim=-1), num_samples=1) generated = torch.cat( (generated, next_token.unsqueeze(0)), dim=1)  This works for the first iteration, but then I get an error for the next iteration:  File "/Users/shamoon/Sites/wordblot/packages/ml-server/generator.py", line 143, in sample_sequence past=past_outputs File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__ result = self.forward(*input, **kwargs) File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 601, in forward output_hidden_states=output_hidden_states, File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__ result = self.forward(*input, **kwargs) File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 470, in forward position_embeds = self.wpe(position_ids) File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__ result = self.forward(*input, **kwargs) File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward self.norm_type, self.scale_grad_by_freq, self.sparse) File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/functional.py", line 1724, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) IndexError: index out of range in self  Is there something I'm doing wrong?

Answer: I did:  outputs, past_outputs = self.models[model_name]( context, past=past_outputs ) context = next_token.unsqueeze(0)
Question: I want to use the FileUpload widget in jupyter lab. I have the following lines of code in my notebook cell: uploader = widgets.FileUpload() uploader  In jupyter notebook, the output of the cell is a clickable button that I can use to upload a file. In jupyter lab, the output is the following : FileUpload(value={}, description='Upload')  Here's the info on the uploader object : Type: FileUpload String form: FileUpload(value={}, description='Upload') File: ~/miniconda3/envs/fastai2/lib/python3.7/site-packages/ipywidgets/widgets/widget_upload.py  Is it possible to make this widget work on jupyter lab? And if so how should I proceed ?

Answer: If you're using jupyterlab out the box, it doesn't have ipywidgets enabled by default, you need to rebuild it after enabling the extension. Follow the steps from here:  Install nodeJS pip install ipywidgets jupyter nbextension enable --py widgetsnbextension jupyter labextension install @jupyter-widgets/jupyterlab-manager (may need to restart your lab)  It says that newer Jupyterlab has it enabled, but I still had troubles with it, depending on the platform. Manual install is usually the way to go.
Question: I have followed the docs about Oauth2 but it does not describe the proccess to add client id and secret https://fastapi.tiangolo.com/advanced/security/oauth2-scopes/ and what this does class UserInDB(User): hashed_password: str  from the original example

Answer: In documentation it uses OAuth2PasswordRequestForm to authenticate the user this class has basically 6 different Fields, grant_type: str = Form(None, regex="password"), username: str = Form(...), password: str = Form(...), scope: str = Form(""), client_id: Optional[str] = Form(None), client_secret: Optional[str] = Form(None),  So you can add client_id and client_secret,if you are interested Repository here. But i usally prefer authlib, it saves so much time makes it easier. Here is a complete example of how you can create a OAuth with authlib First create a OAuth Client from authlib.integrations.starlette_client import OAuth from starlette.config import Config config = Config('.env') # read config from .env file oauth = OAuth(config) oauth.register( name='google', server_metadata_url='https://accounts.google.com/.well-known/openid-configuration', client_kwargs={ 'scope': 'openid email profile' } )  We don't need to add client_id and client_secret here, because they are in .env file. You are not supposed to hard code them in the code in real products.Google has an OpenID discovery endpoint, we can use this URL for server_metadata_url. Authlib will automatically fetch this server_metadata_url to configure the OAuth client for you. Now we will create a FastAPI application to define a login route. from fastapi import FastAPI, Request from starlette.middleware.sessions import SessionMiddleware app = FastAPI() app.add_middleware(SessionMiddleware, secret_key="secret-string")  We need this SessionMiddleware, because Authlib will use request.session to store temporary codes and states. The below code which is /login endpoint, will redirect you to Google account website. @app.route('/login') async def login(request: Request): redirect_uri = request.url_for('auth') return await oauth.google.authorize_redirect(request, redirect_uri  When you grant access from Google website, Google will redirect back to your given redirect_uri, which is request.url_for('auth'): @app.route('/auth') async def auth(request: Request): token = await oauth.google.authorize_access_token(request) user = await oauth.google.parse_id_token(request, token) return user  The above code will obtain a token which contains access_token and id_token. An id_token contains user info, we just need to parse it to get the login user's information. Sources: Authlib-FastAPI-Google-Login Also if you still wanna use Pure FastAPI check this link FastAPI OAuth2PasswordRequestForm
Question: I got this error when I was trying to have a model registered in the model registry. Could someone help me? RestException: INVALID_PARAMETER_VALUE: Unsupported URI './mlruns' for model registry store. Supported schemes are: ['postgresql', 'mysql', 'sqlite', 'mssql']. See https://www.mlflow.org/docs/latest/tracking.html#storage for how to setup a compatible server. 

Answer: Mlflow required DB as datastore for Model Registry So you have to run tracking server with DB as backend-store and log model to this tracking server. The easiest way to use DB is to use SQLite. mlflow server \ --backend-store-uri sqlite:///mlflow.db \ --default-artifact-root ./artifacts \ --host 0.0.0.0  And set MLFLOW_TRACKING_URI environment variable to http://localhost:5000 or mlflow.set_tracking_uri("http://localhost:5000")  After got to http://localhost:5000 and you can register a logged model from UI or from the code.
Question: I have the function that generates character images from a font file using PIL. For the current example, it generates a white background image and a red character text. What I want now is that instead of pure red or any other color I can generate a gradient color. Is this possible with my current code? I have seen this  post but it didn't help me. Edit 1: Currently, I am generating English alphabet images from font files using PIL. The fonts variable in my code has N number of ".ttf" files. lets suppose N=3 all in different styles e.g. style1, style2, style3. My current code will always generate these N different styles with fixed white background and fixed red character color. As shown in the below figure.    Instead of red color for the characters, I would like to apply gradients for each style. i.e. all characters in style1 font images should have the same gradient, style 2 font style should have a different gradient from style1 characters but should be the same for all of its characters and so on. As shown below (styles are different from the above images. Its just for demonstration of what I want).    My code so far: fonts = glob.glob(os.path.join(fonts_dir, '*.ttf')) for font in fonts: image = Image.new('RGB', (IMAGE_WIDTH, IMAGE_HEIGHT), color='white') font = ImageFont.truetype(font, 150) drawing = ImageDraw.Draw(image) w, h = drawing.textsize(character, font=font) drawing.text( ((IMAGE_WIDTH-w)/2, (IMAGE_HEIGHT-h)/2), character, fill='red', font=font ) image.save(file_path, 'PNG') 

Answer: One fairly easy way of doing it is to draw the text in white on a black background and then use that as the alpha/transparency channel over a background with a gradient. Here's a background gradient:  #!/usr/bin/env python3 from PIL import Image, ImageDraw, ImageFont w, h = 400, 150 image = Image.open('gradient.jpg').rotate(90).resize((w,h)) font = ImageFont.truetype('/System/Library/Fonts/MarkerFelt.ttc', 80) # Create new alpha channel - solid black alpha = Image.new('L', (w,h)) draw = ImageDraw.Draw(alpha) draw.text((20,10),'Some Text',fill='white',font=font) alpha.save('alpha.png') # Use text cutout as alpha channel for gradient image image.putalpha(alpha) image.save('result.png')  The alpha.png looks like this:  And the result.png looks like this:  Note that the area around the text is transparent. but you can easily paste it onto a white or black background. So, say you wanted the background yellow, add the following to the bottom of the code above: solid = Image.new('RGB', (w,h), 'yellow') solid.paste(image,image) solid.save('result2.png')
Question: I'm using argparse and I'd like to specify the positional argument with a dash in it. argparse seems to let me do this. Indeed, it shows up in the Namespace of parse_args(), but I cannot figure out how to reference the corresponding value. Here's a minimal example (notice the dash in 'a-string'): #!/usr/bin/env python3 import argparse parser = argparse.ArgumentParser() parser.add_argument('a-string', help='A string') args = parser.parse_args() # AttributeError: 'Namespace' object has no attribute 'a_string' #print("Argument was: " + args.a_string) # TypeError: 'Namespace' object is not subscriptable #print("Argument was: " + args['a-string']) # AttributeError: 'Namespace' object has no attribute 'a' #print("Argument was: " + args.a-string) # I give up. Ask StackOverflow.  I initially thought to address this with the dest argument to add_argument, but I get "ValueError: dest supplied twice for positional argument" if I add dest to a positional argument. How do I reference the value of this positional argument?

Answer: The arguments are given as attributes of the namespace object returned by parse_args(). But identifiers (including attributes) can't have a hyphen in them (at least not if you want to access them directly), because it's a minus sign, and that means something else. If you want your argument to be named "a-string" to the user but accessible in your code as a_string (which was one of your attempted approaches) you can use the metavar argument to specify how it's described to the user. parser.add_argument('a_string', help='A string', metavar='a-string') args = parser.parse_args() print(args.a_string)  The argument will appear in the usage information as: positional arguments: a-string A string  https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument
Question: I know a little of Python and more than a year ago I wrote a small script, using pipenv to manage the dependencies. The old platform was Windows 7, the current platform is Windows 10. At that time I probably had Python 3.7 installed, now I have 3.8.3 but running: pipenv install  Complained that: Warning: Python 3.7 was not found on your system… Neither 'pyenv' nor 'asdf' could be found to install Python. You can specify specific versions of Python with: $ pipenv --python path\to\python  This is the Pipfile [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] python-ldap = {path = "./dependencies/python_ldap-3.1.0-cp37-cp37m-win_amd64.whl"} requests = "~=2.0" mysqlclient = "~=1.0" [dev-packages] [requires] python_version = "3.7"  I manually edited that last line to allow 3.8, but how do I properly fix that? I think 3.7 should be a minimum requirement — well, the script is so simple that I think even 3.0 should work.

Answer: You can download Python 3.7 from the official site - https://www.python.org/downloads/