eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Having recently upgraded a Django project from 2.x to 3.x, I noticed that the mysql.connector.django backend (from mysql-connector-python) no longer works. The last version of Django that it works with is 2.2.11. It breaks with 3.0. I am using mysql-connector-python==8.0.19. When running manage.py runserver, the following error occurs: django.core.exceptions.ImproperlyConfigured: 'mysql.connector.django' isn't an available database backend. Try using 'django.db.backends.XXX', where XXX is one of: 'mysql', 'oracle', 'postgresql', 'sqlite3'  I am aware that this is not an official Django backend but I have to use it on this project for reasons beyond my control. I am 80% sure this is an issue with the library but I'm just looking to see if there is anything that can be done to resolve it beyond waiting for an update. UPDATE: mysql.connector.django now works with Django 3+.

Answer: For Django 3.0 and Django 3.1 I managed to have it working with mysql-connector-python 8.0.22. See this https://dev.mysql.com/doc/relnotes/connector-python/en/news-8-0-22.html.
Question: I'm trying to install virtualenvwrapper (not pyenv-virtualenvwrapper) in my macOS (using zsh). I'm using pyenv to mange multiple python versions and pipx to install CLI stuff. I'm using Python 3.8.1  $ pyenv versions system 2.7.17 * 3.8.1 (set by /Users/my_user/.pyenv/version)  I installed virtualenvwrapper with pipx $ pipx install virtualenvwrapper $ pipx list venvs are in /Users/my_user/.local/pipx/venvs apps are exposed on your $PATH at /Users/my_user/.local/bin package sshuttle 0.78.5, Python 3.8.1 - sshuttle package virtualenv 20.0.15, Python 3.8.1 - virtualenv package virtualenvwrapper 4.8.4, Python 3.8.1 - virtualenvwrapper.sh - virtualenvwrapper_lazy.sh  and I inserted in my .zshrc the following lines: export WORKON_HOME=$HOME/.virtualenvs source /Users/my_user/.local/pipx/venvs/virtualenvwrapper/bin/virtualenvwrapper.sh export PIP_VIRTUALENV_BASE=$WORKON_HOME  But when I launch the shell I get the following error: /Users/my_user/.pyenv/versions/3.8.1/bin/python: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper') virtualenvwrapper.sh: There was a problem running the initialization hooks. If Python could not import the module virtualenvwrapper.hook_loader, check that virtualenvwrapper has been installed for VIRTUALENVWRAPPER_PYTHON=/Users/my_user/.pyenv/shims/python and that PATH is set properly. $  How can I fix this problem?

Answer: Fixed specifying a specific VIRTUALENVWRAPPER_PYTHON without pointing to the shim export WORKON_HOME=$HOME/.virtualenvs export VIRTUALENVWRAPPER_PYTHON=/Users/my_user/.local/pipx/venvs/virtualenvwrapper/bin/python3.8 source /Users/my_user/.local/pipx/venvs/virtualenvwrapper/bin/virtualenvwrapper.sh
Question: Torch documentation says use pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html  to install the latest version of PyTorch. This works when I do it manually but when I add it to req.txt and do pip install -r req.txt, it fails and says ERROR: No matching distribution. Edit: adding the whole line from req.txt and error here. torch==1.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.htmltorch==1.4.0+cpu  ERROR: Could not find a version that satisfies the requirement torch==1.4.0+cpu (from -r requirements.txt (line 1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0) ERROR: No matching distribution found for torch==1.4.0+cpu (from -r requirements.txt (line 1)) 

Answer: Add --find-links in requirements.txt before torch --find-links https://download.pytorch.org/whl/torch_stable.html torch==1.2.0+cpu  Source: https://github.com/pytorch/pytorch/issues/29745#issuecomment-553588171
Question: I am having a .rtf file and I want to read the file and store strings into list using python3 by using any package but it should be compatible with both Windows and Linux. I have tried striprtf but read_rtf is not working. from striprtf.striprtf import rtf_to_text from striprtf.striprtf import read_rtf rtf = read_rtf("file.rtf") text = rtf_to_text(rtf) print(text)  But in this code, the error is: cannot import name 'read_rtf' Please can anyone suggest any way to get strings from .rtf file in python3?

Answer: Have you tried this? with open('yourfile.rtf', 'r') as file: text = file.read() print(text)  For a super large file, try this: with open("yourfile.rtf") as infile: for line in infile: do_something_with(line)
Question:  I am using Google Cloud Function using Python. Several other functions are in production. However, for this, I have additionally created a custom Python package that is available on github as a private repo. I need to install the package in the Google Function  WHAT I HAVE DONE  I run the Google Function in local using functions-framework I have a requirements.txt which has a link to the package. This is done by adding the following line to requirements.txt:   -e git+https://github.com/<repo>/<project>#egg=<project>   I run pip install -r requirements.txt. And the package is successfully installed. Now in the python code of the function using import <pkg-name> works and I am able to access all the functions.  CHALLENGES WHEN PUSHING THE FUNCTION TO THE CLOUD  As per the documentation, to push the Cloud function to Google Cloud, I issue the command:  gcloud functions \ deploy <function-name> \ --trigger-http \ --entry-point <entry-function> \ --runtime python37 \ --project=<my-project>  As expected this gives an error because it does not have access to the private repo in git.  I created a Google Cloud Repository and linked it to the git repo, hoping that in some way I could specify that in the requirements.txt. Just do not know how. I tried setting environment variables for username and password (not a good idea, I agree) in Google Cloud Function and specify them in the requirements.txt as:   -e git+https://${AUTH_USER}:${AUTH_PASSWORD}@github.com/<repo>/<project>#egg=<project>  That too gave an error. Any help or direction will be greatly appreciated. 

Answer: You can not access the private repo from cloud function. According to the official documentation:  " Using private dependencies Dependencies are installed in a Cloud Build environment that does not provide access to SSH keys. Packages hosted in repositories that require SSH-based authentication must be vendored and uploaded alongside your project's code, as described in the previous section. You can use the pip install command with the -t DIRECTORY flag to copy private dependencies into a local directory before deploying your app, as follows: Copy your dependency into a local directory: pip install -t DIRECTORY DEPENDENCY  Add an empty init.py file to the DIRECTORY directory to turn it into a module. Import from this module to use your dependency: import DIRECTORY.DEPENDENCY  "  Specifying dependencies in Python
Question: I am using JupyterLab with light theme and when I exported my notebook as HTML I saw this:  What I am expecting to see is something like this:  any ideas of the setting ?

Answer: I had the exact same issue. After a couple hours debugging I realized it had to do (for me at least) with the jupyter-theme library. I had a dark theme installed, and I think nbconverter uses whichever settings your jupyter is also using, so the dark settings were affecting the html conversion.  Solution was simply to restore defaults with: $ jt -r If that doesn't work, then refer to this thread: https://github.com/dunovank/jupyter-themes/issues/86
Question: I like to use Plotly to visualize everything, I'm trying to visualize a confusion matrix by Plotly, this is my code: def plot_confusion_matrix(y_true, y_pred, class_names): confusion_matrix = metrics.confusion_matrix(y_true, y_pred) confusion_matrix = confusion_matrix.astype(int) layout = { "title": "Confusion Matrix", "xaxis": {"title": "Predicted value"}, "yaxis": {"title": "Real value"} } fig = go.Figure(data=go.Heatmap(z=confusion_matrix, x=class_names, y=class_names, hoverongaps=False), layout=layout) fig.show()  and the result is   How can I show the number inside corresponding cell instead of hovering, like this

Answer: You can use annotated heatmaps with ff.create_annotated_heatmap() to get this:  Complete code: import plotly.figure_factory as ff z = [[0.1, 0.3, 0.5, 0.2], [1.0, 0.8, 0.6, 0.1], [0.1, 0.3, 0.6, 0.9], [0.6, 0.4, 0.2, 0.2]] x = ['healthy', 'multiple diseases', 'rust', 'scab'] y = ['healthy', 'multiple diseases', 'rust', 'scab'] # change each element of z to type string for annotations z_text = [[str(y) for y in x] for x in z] # set up figure fig = ff.create_annotated_heatmap(z, x=x, y=y, annotation_text=z_text, colorscale='Viridis') # add title fig.update_layout(title_text='<i><b>Confusion matrix</b></i>', #xaxis = dict(title='x'), #yaxis = dict(title='x') ) # add custom xaxis title fig.add_annotation(dict(font=dict(color="black",size=14), x=0.5, y=-0.15, showarrow=False, text="Predicted value", xref="paper", yref="paper")) # add custom yaxis title fig.add_annotation(dict(font=dict(color="black",size=14), x=-0.35, y=0.5, showarrow=False, text="Real value", textangle=-90, xref="paper", yref="paper")) # adjust margins to make room for yaxis title fig.update_layout(margin=dict(t=50, l=200)) # add colorbar fig['data'][0]['showscale'] = True fig.show()
Question: I have bipartite graph with nodes such as(a1,a2,...a100, m1,m2,...). I want to find the induced subgraph for certain nodes say(a1,a2 and a10). I can do this by using networkx.ego_graph, but it takes one vertex at one time and returns the induced graph. I want to know if there is any way to do this at once for all the nodes that i am interested in and then select the one that is largest.

Answer: For the general case, the ego graph can be obtained using nx.ego_graph.  Though in your specific case, it looks like you want to find the largest induced ego graph in the network. For that you can first find the node with a highest degree, and then obtain its ego graph.  Let's create an example bipartite graph: import networkx as nx B = nx.Graph() B.add_nodes_from([1, 2, 3, 4, 5, 6], bipartite=0) B.add_nodes_from(['a', 'b', 'c', 'j', 'k'], bipartite=1) B.add_edges_from([(1, 'a'), (1, 'b'), (2, 'b'), (2, 'c'), (3, 'c'), (4, 'a'), (2, 'b'), (3, 'a'), (5, 'k'), (6, 'k'), (6, 'j')]) rcParams['figure.figsize'] = 12, 6 nx.draw(B, node_color='lightblue', with_labels=True)   And as mentioned in the question, say we want to select among the following list of nodes: l = [1,'a',6]  It looks like you want to select the one that has the highest centrality degree among these. For that you could do: deg_l = {i:B.degree(i) for i in l} highest_centrality_node = max(deg_l.items(), key=lambda x: x[1])[0]  Now we could plot the corresponding ego_graph with: ego_g = nx.ego_graph(B, highest_centrality_node) d = dict(ego_g.degree) nx.draw(ego_g, node_color='lightblue', with_labels=True, nodelist=d, node_size=[d[k]*300 for k in d])
Question: I am currently running a conda environment with flask-wtf version 0.14.2 and wtforms version 2.21 and I have trouble solving this ImportError: cannot import name 'url_encode' from 'werkzeug' The following code is the complete traceback. Traceback (most recent call last): File "run.py", line 1, in <module> from flaskblog import app File "/Users/justinding/Desktop/test/test_wesite/flaskblog/__init__.py", line 10, in <module> from flaskblog import routes File "/Users/justinding/Desktop/test/test_wesite/flaskblog/routes.py", line 4, in <module> from flaskblog.forms import RegistrationForm,LoginForm File "/Users/justinding/Desktop/test/test_wesite/flaskblog/forms.py", line 1, in <module> from flask_wtf import FlaskForm File "/opt/anaconda3/envs/smartbox/lib/python3.7/site-packages/flask_wtf/__init__.py", line 17, in <module> from .recaptcha import * File "/opt/anaconda3/envs/smartbox/lib/python3.7/site-packages/flask_wtf/recaptcha/__init__.py", line 2, in <module> from .fields import * File "/opt/anaconda3/envs/smartbox/lib/python3.7/site-packages/flask_wtf/recaptcha/fields.py", line 3, in <module> from . import widgets File "/opt/anaconda3/envs/smartbox/lib/python3.7/site-packages/flask_wtf/recaptcha/widgets.py", line 5, in <module> from werkzeug import url_encode ImportError: cannot import name 'url_encode' from 'werkzeug' (/opt/anaconda3/envs/smartbox/lib/python3.7/site-packages/werkzeug/__init__.py) 

Answer: Setting werkzeug==0.16.1 in your requirements file fixes it. The issue is with the 1.0.0 version
Question: I had a different question here, but realized it simplifies to this: How do you detect when a client disconnects (closes their page or clicks a link) from a page (in other words, the socket connection closes)? I want to make a chat app with an updating user list, and I’m using Flask on Python. When the user connects, the browser sends a socket.emit() with an event and username passed in order to tell the server a new user exists, after which the server will message all clients with socket.emit(), so that all clients will append this new user to their user list. However, I want the clients to also send a message containing their username to the server on Disconnect. I couldn’t figure out how to get the triggers right. Note: I’m just using a simple html file with script tags for the page, I’m not sure how to add a JS file to go along with the page, though I can figure it out if it’s necessary for this.

Answer: Figured it out. socket.on('disconnect') did turn out to be right, however by default it pings each user only once a minute or so, meaning it took a long time to see the event.
Question: Given a python list of bytes values: # actual str values un-important [ b'foo', b'bar', b'baz', ... ]  How can the list be broken into chunks where each chunk has the maximum memory size below a certain ceiling? For example: if the ceiling were 7 bytes, then the original list would be broken up into a list of lists [ [b'foo', b'bar'], # sublist 0 [b'baz'], # sublist 1 ... ]  And each sublist would be at most 7 bytes, based on accumulated length of the list's contents.  Note: each sub-list should be maximally packed, in the order of the original list. In the example above the first 2 str values were grouped because it is the maximum possible under the 7 byte limit. Thank you in advance for your consideration and response.

Answer: This solution is with functools.reduce. l = [b'abc', b'def', b'ghi', b'jklm', b'nopqrstuv', b'wx', b'yz'] reduce(lambda a, b, size=7: a[-1].append(b) or a if a and sum(len(x) for x in a[-1]) + len(b) <= size else a.append([b]) or a, l, [])  a is an empty list and b is an item from the original list. if a and sum(len(x) for x in a[-1]) + len(b) <= size check if a is not empty and sum of length of bytes in the last appended list and length of b is not exceeding size.  a[-1].append(b) or a append b to the last appended list of a and return a if the condition is True.  a.append([b]) or a make a list with b and append the new list to a and return a Output; [[b'abc', b'def'], [b'ghi', b'jklm'], [b'nopqrstuv'], [b'wx', b'yz']]
Question: I use tqdm module in Jupyter Notebook. And let's say I have the following piece of code with a nested for loop. import time from tqdm.notebook import tqdm for i in tqdm(range(3)): for j in tqdm(range(5)): time.sleep(1)  The output looks like this: 100%|██████████| 3/3 [00:15<00:00, 5.07s/it] 100%|██████████| 5/5 [00:10<00:00, 2.02s/it] 100%|██████████| 5/5 [00:05<00:00, 1.01s/it] 100%|██████████| 5/5 [00:05<00:00, 1.01s/it]  Is there any option, how to show only current j progress bar during the run? So, the final output after finishing the iteration would look like this? 100%|██████████| 3/3 [00:15<00:00, 5.07s/it] 100%|██████████| 5/5 [00:05<00:00, 1.01s/it] 

Answer: You can use leave param when create progress bar. Something like this: import time from tqdm import tqdm for i in tqdm(range(3)): for j in tqdm(range(5), leave=bool(i == 2)): time.sleep(1)
Question: PS E:\Python and Data Science\PythonDatabase> conda activate base conda : The term 'conda' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + conda activate base + ~~~~~ + CategoryInfo : ObjectNotFound: (conda:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException PS E:\Python and Data Science\PythonDatabase> & C:/Users/Lenovo/Anaconda3/python.exe "e:/Python and Data Science/PythonDatabase/CourseHelper.py" Hello World PS E:\Python and Data Science\PythonDatabase> 

Answer: You can set "python.terminal.activateEnvironment": false in your settings to deactivate activation of your environment. Alternatively, you can set "python.condaPath" to where conda exists so the extension can use conda appropriately.
Question: I wish to place a python program on GitHub and have other people download and run it on their computers with assorted operating systems. I am relatively new to python but have used it enough to have noticed that getting the assorted versions of all the included modules to work together can be problematic. I just discovered the use of requirements.txt (generated with pipreqs and deployed with the command pip install -r /path/to/requirements.txt) but was very surprised to notice that requirements.txt does not actually state what version of python is being used so obviously it is not the complete solution on its own. So my question is: what set of specifications/files/something-else is needed to ensure that someone downloading my project will actually be able to run it with the fewest possible problems. EDIT: My plan was to be guided by whichever answer got the most upvotes. But so far, after 4 answers and 127 views, not a single answer has even one upvote. If some of the answers are no good, it would be useful to see some comments as to why they are no good.

Answer: Have you considered setting up a setup.py file? It's a handy way of bundling all of your... well setup into a single location. So all your user has to do is A) clone your repo and B) run pip install . to run the setup.py There's a great stack discussion about this. As well as a handle example written by the requests guy. This should cover most use cases. Now if you want to make it truly distributable then you'll want to look into setting it up in PyPi, the official distribution hub. Beyond that if you're asking how to make a program "OS independent" there isn't a one size fits all. It depends on what you are doing with your code. Requires researching how your particular code interacts with those OS's etc.
Question: I have three columns about year, month and day. How can I use these to create date in PySpark?

Answer: You can use concat_ws() to concat columns with - and cast to date. #sampledata df.show() #+----+-----+---+ #|year|month|day| #+----+-----+---+ #|2020| 12| 12| #+----+-----+---+ from pyspark.sql.functions import * df.withColumn("date",concat_ws("-",col("year"),col("month"),col("day")).cast("date")).show() +----+-----+---+----------+ |year|month|day| date| +----+-----+---+----------+ |2020| 12| 12|2020-12-12| +----+-----+---+----------+ #dynamic way cols=["year","month","day"] df.withColumn("date",concat_ws("-",*cols).cast("date")).show() #+----+-----+---+----------+ #|year|month|day| date| #+----+-----+---+----------+ #|2020| 12| 12|2020-12-12| #+----+-----+---+----------+ #using date_format,to_timestamp,from_unixtime(unix_timestamp) functions df.withColumn("date",date_format(concat_ws("-",*cols),"yyyy-MM-dd").cast("date")).show() df.withColumn("date",to_timestamp(concat_ws("-",*cols),"yyyy-MM-dd").cast("date")).show() df.withColumn("date",to_date(concat_ws("-",*cols),"yyyy-MM-dd")).show() df.withColumn("date",from_unixtime(unix_timestamp(concat_ws("-",*cols),"yyyy-MM-dd"),"yyyy-MM-dd").cast("date")).show() #+----+-----+---+----------+ #|year|month|day| date| #+----+-----+---+----------+ #|2020| 12| 12|2020-12-12| #+----+-----+---+----------+
Question: I would like to create a new columns in a pandas DataFrame just like I would do using a python f-Strings or format function. Here is an example: df = pd.DataFrame({"str": ["a", "b", "c", "d", "e"], "int": [1, 2, 3, 4, 5]}) print(df) str int 0 a 1 1 b 2 2 c 3 3 d 4 4 e 5  I would like to obtain:   str int concat 0 a 1 a-01 1 b 2 b-02 2 c 3 c-03 3 d 4 d-04 4 e 5 e-05  So something like: concat = f"{str}-{int:02d}"  but directly with elements of pandas columns. I imagine the solution is using pandas map, apply, agg but nothing successful. Many thanks for your help. 

Answer: Use lsit comprehension with f-strings: df['concat'] = [f"{a}-{b:02d}" for a, b in zip(df['str'], df['int'])]  Or is possible use apply: df['concat'] = df.apply(lambda x: f"{x['str']}-{x['int']:02d}", axis=1)  Or solution from comments with Series.str.zfill: df["concat"] = df["str"] + "-" + df["int"].astype(str).str.zfill(2)   print (df) str int concat 0 a 1 a-01 1 b 2 b-02 2 c 3 c-03 3 d 4 d-04 4 e 5 e-05
Question: I want to get count with Motor's diver but I got this error. AttributeError: 'AsyncIOMotorCursor' object has no attribute 'count' This is my code: await MOTOR_CURSOR.users.find().count() 

Answer: MotorCollection.find() returns AsyncIOMotorCursor and it does not have count method. Instead, you should call MotorCollection.count_documents() instead.  await db.users.count_documents({'x': 1})  Also worth noting that what you're referring to as MOTOR_CURSOR is MotorDatabase instance, it would preferable to call it a db instance instead of a cursor to avoid confusion.
Question: I'm a new to factory_boy module. In my code, I import factory and then used this import to access the fuzzy attribute with factory.fuzzy then it throws error module 'factory' has no attribute 'fuzzy'. I solved this problem by again importing like this  import factory from factory import fuzzy by doing so there were no errors. What is the reason for this!

Answer: Why this happens? When you import a Python module (your import factory), you can then access directly what is declared in that module (e.g factory.Factory): all symbols declared in the module are automatically exported. However, if a nested module is not imported in its parent, you have to import it directly. Here, factory.Factory is available, because factory/__init__.py contains: from .base import Factory  => When you type factory.Factory, Python looks up the symbol named Factory in factory/__init__.py, which is (per the above line) a reference to the Factory class defined in factory/base.py. Since there is no line with from . import fuzzy in factory/__init__.py, Python cannot load it this way. But why don't you add this line? Other modules in the factory_boy package have dependencies on third-party packages; for instance, factory.django imports Django. If factory/__init__.py contained the from . import django line (required to have factory.django available from import factory), every program that runs import factory would require to have Django installed. In order to allow users of the package to decide what they depend on, the choice was made to not add those direct imports at the package top-level when possible — this allows for future versions to add external dependencies without breaking existing code.
Question: I have a df like this: frame = pd.DataFrame({'a' : ['a,b,c', 'a,c,f', 'b,d,f','a,z,c']})  And a list of items: letters = ['a','c']  My goal is to get all the rows from frame that contain at least the 2 elements in letters I came up with this solution: for i in letters: subframe = frame[frame['a'].str.contains(i)]  This gives me what I want, but it might not be the best solution in terms of scalability. Is there any 'vectorised' solution? Thanks

Answer: I would build a list of Series, and then apply a vectorized np.all: contains = [frame['a'].str.contains(i) for i in letters] resul = frame[np.all(contains, axis=0)]  It gives as expected:  a 0 a,b,c 1 a,c,f 3 a,z,c