eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: We're upgrading to use BeeWare's Briefcase 0.3.1 for packaging, which uses pyproject.toml instead of setup.py to specify how to package, including which dependencies to include in a package. Here's a minimal example of a pyproject.toml for briefcase: [tool.briefcase.app.exampleapp] formal_name = "exampleapp" description = "something" requires = ['PyQt5', 'qtconsole'] sources = ['exampleapp']  We'd like to access the list of requires from setup.py, so we wouldn't have to replicate it in both files, and keep them in sync. We're not ready to switch away from setuptools, this is only for packaging. The alternative is of course to let setup.py auto-generate the pyproject.toml file, but that seems a little backwards to the intention with PEP 518.

Answer: This answer might be outdated. I do not have time to investigate right now. I recommend checking the briefcase resources for more up-to-date information. For example this section of the doc might be relevant: https://briefcase.readthedocs.io/en/latest/reference/configuration.html#pep621-compatibility  As far as I can tell, briefcase isn't actually PEP 517 compatible (at least not by default). It uses a pyproject.toml file, but doesn't fill up the [build-system] section, so it should be possible to set an actual PEP 517 build backend in that file without causing conflict. pyproject.toml [build-system] build-backend = 'setuptools.build_meta' requires = [ 'setuptools', 'toml', ] [tool.briefcase.app.exampleapp] formal_name = 'exampleapp' description = 'something' requires = ['PyQt5', 'qtconsole'] sources = ['exampleapp']  setup.py #!/usr/bin/env python3 import pathlib import pkg_resources import setuptools import toml def _parse_briefcase_toml(pyproject_path, app_name): pyproject_text = pyproject_path.read_text() pyproject_data = toml.loads(pyproject_text) briefcase_data = pyproject_data['tool']['briefcase'] app_data = briefcase_data['app'][app_name] setup_data = { 'name': pkg_resources.safe_name(app_data['formal_name']), 'version': briefcase_data['version'], 'install_requires': app_data['requires'], # ... } return setup_data def _setup(): app_name = 'exampleapp' pyproject_path = pathlib.Path('pyproject.toml') setup_data = _parse_briefcase_toml(pyproject_path, app_name) setuptools.setup(**setup_data) if __name__ == '__main__': _setup()  Then pip and other PEP 517-compatible frontends should be able to build and install the project by delegating to setuptools while taking care to correctly setup a build environment containing both setuptools and toml. I guess it would be also possible to let briefcase handle the parsing of the pyproject.toml file (maybe with briefcase.config.parse_config(...)) but it's not documented so I don't know how stable these APIs are.
Question: import logging from confluent_kafka import Producer import os logger = logging.getLogger("main") BOOTSTRAP_SERVERS = os.environ['BOOTSTRAP_SERVERS'] APPLICATION_ID = os.getenv('APPLICATION_ID', default = "nke-data-source") RECONNECT_BACKOFF_MS = os.getenv('RECONNECT_BACKOFF_MS', default = 1000) REQUEST_TIMEOUT_MS = os.getenv('REQUEST_TIMEOUT_MS', default = 40000) ACKS = os.getenv('ACKS', default = "all") RETRIES = os.getenv('RETRIES', default = 15) RETRY_BACK_OFF = os.getenv('RETRY_BACK_OFF', default = 1000) MAX_IN_FLIGHT_REQUESTS = os.getenv('MAX_IN_FLIGHT_REQUESTS', default = 1) topic = os.getenv('OUTBOUND_TOPIC', default = "tti-nke-raw") p = Producer({'bootstrap.servers': BOOTSTRAP_SERVERS, 'client.id': APPLICATION_ID, 'reconnect.backoff.ms': RECONNECT_BACKOFF_MS, 'request.timeout.ms': REQUEST_TIMEOUT_MS, 'acks': ACKS, 'retries': RETRIES, 'retry.backoff.ms': RETRY_BACK_OFF, 'max.in.flight.requests.per.connection': MAX_IN_FLIGHT_REQUESTS, 'compression.type': "lz4"}) def send(key, event): try: logger.info("Sending key: [{0}] value: [{1}]".format(key, event)) p.produce(topic=topic, value=event.encode('utf-8'), key=key) except Exception: logger.error("error sending events to kafka", exc_info=True)  Error:-  Traceback (most recent call last): BufferError: Local: Queue full File "/app/sender.py", line 30, in send p.produce(topic=topic, value=event.encode('utf-8'), key=key)  Can anyone help me in this as I'am new in python

Answer: This Queue is something implemented in the librdkafka library (which confluent_kafka is binding to) There is an inner Queue for the produce that takes the producer delivery report and waits for the produce to deal with them (mostly doing nothing), but you need to trigger this mechanism of going through the queue, which can be executed by calling poll You should call producer.poll(0) after every call of produce so change: p.produce(topic=topic, value=event.encode('utf-8'), key=key)  to: p.produce(topic=topic, value=event.encode('utf-8'), key=key) p.poll(0)  Which will trigger the queue cleaning, don't worry about performance because this is a very simple function that doesn't really do much as the author of librdkafka wrote:  poll() is cheap to call, it will not have a performance impact, so please add it to your producer loop.  basically it does:  call poll() at regular intervals to serve the producer's delivery report callbacks.  consider to read about this in this Issue too
Question: I have the Polygon data from the States from the USA from the website arcgis and I also have an excel file with coordinates of citys. I have converted the coordinates to geometry data (Points). Now I want to test if the Points are in the USA. Both are dtype: geometry. I thought with this I can easily compare, but when I use my code I get for every Point the answer false. Even if there are Points that are in the USA. The code is: import geopandas as gp import pandas as pd import xlsxwriter import xlrd from shapely.geometry import Point, Polygon df1 = pd.read_excel('PATH') gdf = gp.GeoDataFrame(df1, geometry= gp.points_from_xy(df1.longitude, df1.latitude)) US = gp.read_file('PATH') print(gdf['geometry'].contains(US['geometry']))  Does anybody know what I do wrong?

Answer: contains in GeoPandas currently work on a pairwise basis 1-to-1, not 1-to-many. For this purpose, use sjoin. points_within = gp.sjoin(gdf, US, predicate='within')  That will return only those points within the US. Alternatively, you can filter polygons which contain points. polygons_contains = gp.sjoin(US, gdf, predicate='contains')
Question: This message keeps popping out after I used two GPUs simultaneously for two notebooks from the same account for about half an hour (Colab wasn't running for 12 hours): Photo of pop-out message  You cannot currently connect to a GPU due to usage limits in Colab.  It has been about two hours since I last used colab, but the message still pops up. It would be really great if I know for how long does this lasts. I think it could be 12 hours but would want to know from someone who has experienced this. In my case, the GPUs are available, but due to recent excess computing and running one cell for long, I have reached my usage limit for gpus. However, I want to know that after how much waiting will colab let me use its GPUs again. Update: It has been more than 2 days and colab still doesn't allow me to use GPUs. The usage limit message still pops out.

Answer: The usage limit is pretty dynamic and depends on how much/long you use colab. I was able to use the GPUs after 5 days; however, my account again reached usage limit right after 30mins of using the GPUs (google must have decreased it further for my account). The situation really became normal after months of not using colab from that account. My suggestion is to have multiple google accounts for colab, so you could use the other accounts when facing usage limits. Sorry for not replying to the comments.
Question: I have a flask app that uses some enviroment variables, I don't now what has changed but now it doesn't read the variables in the .flaskenv file, already tried changing its name to .env, still not working. This is the .flaskenv file:  FLASK_APP=app FLASK_ENV=development CONSUMER_KEY= CONSUMER_SECRET= ACCESS_KEY= ACCESS_SECRET= SEC_USERNAME= SEC_PASSWORD=  In the app.py file:  from flask import Flask, render_template, url_for, request, redirect, session, g import os s_user = os.environ.get('SEC_USERNAME') s_pass = os.environ.get('SEC_PASSWORD') print(s_user) print(s_pass) if __name__ == "__main__": app.run()  It prints "none".  If i do flask run : Usage: flask run [OPTIONS] Error: Could not locate Flask application. You did not provide the FLASK_APP environment variable. For more information see http://flask.pocoo.org/docs/latest/quickstart/  pip list: (venv) λ pip list Package Version ----------------- ---------- certifi 2020.4.5.2 chardet 3.0.4 click 6.7 Flask 0.12.2 idna 2.9 itsdangerous 0.24 Jinja2 2.10 MarkupSafe 1.0 oauthlib 3.1.0 pip 19.2.3 PySocks 1.7.1 python-dotenv 0.13.0 requests 2.23.0 requests-oauthlib 1.3.0 setuptools 41.2.0 six 1.15.0 tweepy 3.8.0 urllib3 1.25.9 Werkzeug 0.13 

Answer: As written in the flask docs:  If python-dotenv is installed, running the flask command will set environment variables defined in the files .env and .flaskenv. This can be used to avoid having to set FLASK_APP manually every time you open a new terminal, and to set configuration using environment variables similar to how some deployment services work.  make sure to run pip install python-dotenv Edit This flask seems to be very old, try updating it with pip install -U Flask
Question: Recently the ms-python extension (v2020.5.86806) for vscode implements grouping of variables in the debug console/variable explorer. They appear as: <object> > special variables > function variables  Is there a way to disable this behavior? EDIT: Screenshot added: 

Answer: There's no single flag to revert to old behavior, but you can fine-tune it on a per-group basis in your launch.json: { "version": "0.2.0", "configurations": [ { "name": ..., "module": ..., ... "variablePresentation": { "all": "inline", "class": "group", "function": "hide", "protected": ..., "special": ... } } ] }  "all" applies to all groups, and sets the default that can be overridden as needed; other group names are self-explanatory. For values, "group" is the default behavior, "hide" removes those variables entirely, and "inline" places them without grouping. Note that the VSCode JSON schema hasn't been updated to reflect this yet, so you'll get squiggles when editing. It'll still work, though.
Question: I'm trying to be a good citizen and make sure my notebook session is terminated immediately after running even if I'm not sitting at my machine. Is there any code I can run in a notebook cell to achieve this?

Answer: We have a way to do this correctly now: from google.colab import runtime runtime.unassign()
Question: 1st approach: I am trying to make the below code work since morning. I have read many answers here in stackoverflow and tutorials on google about python, I have done 0 progress. Can you help me with this error: Using TensorFlow backend. Traceback (most recent call last): File "source_code_modified.py", line 65, in <module> dict1 = pickle.load(f1,encoding='bytes') TypeError: file must have 'read' and 'readline' attributes  The code that explodes is this part: class load_train_data: os.open("/home/just_learning/Desktop/CNN/datasets/training_data/images", os.O_RDONLY) def __enter__(self): return self def __exit__(self, type, value, traceback): pass class load_test_data: os.open("/home/just_learning/Desktop/CNN/datasets/test_data/images",os.O_RDONLY) def __enter__(self): return self def __exit__(self, type, value, traceback): pass with load_train_data() as f1: dict1 = pickle.load(f1,encoding='bytes')  2nd approach: Ok, I did that, the error is:  Using TensorFlow backend. Traceback (most recent call last): File "source_code_modified.py", line 74, in <module> with open_train_data() as f1: File "source_code_modified.py", line 47, in open_train_data return open('/home/just_learning/Desktop/CNN/datasets/training_data/images','rb') IsADirectoryError: [Errno 21] Is a directory: '/home/just_learning/Desktop/CNN/datasets/training_data/images'  And the code explodes on these points: def open_train_data(): return open('/home/just_learning/Desktop/CNN/datasets/training_data/images','rb') <--- explodes here def open_test_data(): return open('/home/just_learning/Desktop/CNN/datasets/test_data/images','rb') with open_train_data() as f1: dict1 = pickle.load(f1) <--- explodes here  3rd approach: I found this: "IsADirectoryError: [Errno 21] Is a directory: " It is a file and I changed the code to this: def open_train_data(): return os.listdir('/home/just_learning/Desktop/CNN/datasets/training_data/images') def open_test_data(): return os.listdir('/home/just_learning/Desktop/CNN/datasets/test_data/images') with open_train_data() as f1: <-------------- explodes here dict1 = pickle.load(f1) #,encoding='bytes')  and the error is this: Using TensorFlow backend. Traceback (most recent call last): File "source_code_modified.py", line 74, in <module> with open_train_data() as f1: AttributeError: __enter__  and this error I solved it by searching in github/google and was the result depicted on the first approach, that I have posted above...

Answer: Just sharing some stupid mistake for anyone else affected (by stupidity)^^: I had the error TypeError: file must have 'read' and 'readline' attributes because I used the plain text file_path instead of the opened f file object as the parameter of the pickle function. Correct: file_path = 'path/to/filename' with open(file_path , 'rb') as f: dict1 = pickle.load(f)  Wrong: file_path = 'path/to/filename' with open(file_path , 'rb') as f: dict1 = pickle.load(file_path)
Question: I am trying to install python3 virtualenv. I get the following message when I try to run virtualenv. virtualenv Command 'virtualenv' not found, but can be installed with: apt install python3-virtualenv  but if I run install command, I get the following error. apt install python3-virtualenv Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package python3-virtualenv  For python3 -m venv, I get message to install using apt-get install python3-venv but when I try it, I get the same message. sudo apt-get install python3-venv Reading package lists... Done Building dependency tree Reading state information... Done Package python3-venv is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source E: Package 'python3-venv' has no installation candidate  I am running this as root. wget also works.

Answer: AFAIU the latest versions of Ubuntu removed Python2 altogether so Python3 is now just the Python. Try: apt-get update apt-get install python3-virtualenv
Question: I have recently started learning python to develop a predictive model for a research project using machine learning methods. I have a large dataset comprised of both numerical and categorical data. The dataset has lots of missing values. I am currently trying to encode the categorical features using OneHotEncoder. When I read about OneHotEncoder, my understanding was that for a missing value (NaN), OneHotEncoder would assign 0s to all the feature's categories, as such: 0 Male 1 Female 2 NaN  After applying OneHotEncoder: 0 10 1 01 2 00  However, when running the following code:  # Encoding categorical data from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder ct = ColumnTransformer([('encoder', OneHotEncoder(handle_unknown='ignore'), [1])], remainder='passthrough') obj_df = np.array(ct.fit_transform(obj_df)) print(obj_df)  I am getting the error ValueError: Input contains NaN So I am guessing my previous understanding of how OneHotEncoder handles missing values is wrong. Is there a way for me to get the functionality described above? I know imputing the missing values before encoding will resolve this issue, but I am reluctant to do this as I am dealing with medical data and fear that imputation may decrease the predictive accuracy of my model.  I found this question that is similar but the answer doesn't offer a detailed enough solution on how to deal with the NaN values. Let me know what your thoughts are, thanks.

Answer: You will need to impute the missing values before. You can define a Pipeline with an imputing step using SimpleImputer setting a constant strategy to input a new category for null fields, prior to the OneHot encoding: from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder from sklearn.impute import SimpleImputer from sklearn.pipeline import Pipeline import numpy as np categorical_transformer = Pipeline(steps=[ ('imputer', SimpleImputer(strategy='constant', fill_value='missing')), ('encoder', OneHotEncoder(handle_unknown='ignore'))]) preprocessor = ColumnTransformer( transformers=[ ('cat', categorical_transformer, [0]) ])   df = pd.DataFrame(['Male', 'Female', np.nan]) preprocessor.fit_transform(df) array([[0., 1., 0.], [1., 0., 0.], [0., 0., 1.]])
Question: I have a Python Flask app and have lots of environment variables that I need to set when running my app. I normally run my app like so... python3 -m app.py  I would like it so that I can set all the environment variables my app needs so that I do not need to export each time I reopen my terminal. It would be nice if it could be workspace-specific or project-specific.  I know other editors like Pycharm can do similar things and was wondering how to do this in VS Code? 

Answer: For Flask apps, you can create a launch.json configuration that will run your Flask app using VS Code's debugger. VS Code's launch.json supports a number of options, including the setting of environment variables that your Flask app needs. Start with installing the Python extension for VS Code, to add support for "debugging of a number of types of Python applications, which includes Flask. Then, follow the Flask Tutorial from the VS Code docs. Basically, you'll need to create a launch.json file in your workspace's .vscode folder. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "run-flask-app", "type": "python", "request": "launch", "module": "flask", "env": { "FLASK_APP": "/path/to/app.py", "FLASK_ENV": "development", "FLASK_DEBUG": "0", "AAA": "value of AAA env var", "BBB": "value of BBB env var", "CCC": "value of CCC env var" }, "args": [ "run", "--no-debugger", "--no-reload" ], "jinja": true }, ] }  The env portion is where you set all the environment variables. By default, it includes the required FLASK_*-related variables because it uses the flask command line tool for running apps. Here, as an example, I've also set custom AAA and BBB and CCC vars. You can access that from code as normal env vars: @app.route('/test') def root(): aaa = os.environ.get("AAA") bbb = os.environ.get("BBB") ccc = os.environ.get("CCC") print(f'{aaa}, {bbb}, {ccc}') return f'{aaa}, {bbb}, {ccc}'  Then run it from the Debug panel:  Just hit the play button and you should be able to see the printed env values from the Terminal panel or from the browser after accessing the /test endpoint. (If you have multiple terminals, it should appear under the Python Debug Console).  Lastly, by saving the .vscode/launch.json file under the workspace folder, then it's workspace-specific, and it will only affect the runtime environment used to launch your Flask app.
Question: I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is there a more formal doing this? Such as can we calculate the area of non overlapping area between two curves? When I googled for the area between two curves there are many many links but none of them could solve my exact problem. NOTE: The main aim of this plot is to find whether the feature is important or not. So, please suggest me further if I am missing any hidden concepts here. What I am trying to do is set some threshold such as 0.2, if the non-overlapping area > 0.2, then assert that the feature is important, otherwise not. MWE: import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt df = sns.load_dataset('titanic') x0 = df.loc[df['survived']==0,'fare'] x1 = df.loc[df['survived']==1,'fare'] sns.kdeplot(x0,shade=1) sns.kdeplot(x1,shade=1)  Output  Similar links  Fill area of overlap between two normal distributions in seaborn / matplotlib Python: Overlap between two functions (PDF of kde and normal) Fill area between two curves in python 

Answer: Here are my ideas about the computational part of the question:  In order to compare the kde's, they need to be calculated with the same bandwidth. (The default bandwidth depends on the number of x-values, which can be different for both sets.) The intersection of two positive curves is just their minimum. The area of a curve can be approximated via the trapezium rule: np.trapz.  Here are these ideas converted to some example code and illustrating plot: import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt from scipy.stats import gaussian_kde df = sns.load_dataset('titanic') x0 = df.loc[df['survived'] == 0, 'fare'] x1 = df.loc[df['survived'] == 1, 'fare'] kde0 = gaussian_kde(x0, bw_method=0.3) kde1 = gaussian_kde(x1, bw_method=0.3) xmin = min(x0.min(), x1.min()) xmax = max(x0.max(), x1.max()) dx = 0.2 * (xmax - xmin) # add a 20% margin, as the kde is wider than the data xmin -= dx xmax += dx x = np.linspace(xmin, xmax, 500) kde0_x = kde0(x) kde1_x = kde1(x) inters_x = np.minimum(kde0_x, kde1_x) plt.plot(x, kde0_x, color='b', label='No') plt.fill_between(x, kde0_x, 0, color='b', alpha=0.2) plt.plot(x, kde1_x, color='orange', label='Yes') plt.fill_between(x, kde1_x, 0, color='orange', alpha=0.2) plt.plot(x, inters_x, color='r') plt.fill_between(x, inters_x, 0, facecolor='none', edgecolor='r', hatch='xx', label='intersection') area_inters_x = np.trapz(inters_x, x) handles, labels = plt.gca().get_legend_handles_labels() labels[2] += f': {area_inters_x * 100:.1f} %' plt.legend(handles, labels, title='Survived?') plt.title('Fare vs Survived') plt.tight_layout() plt.show()
Question: I am trying to run a project from github , every object counter applications using sort algorithm. I can't run any of them because of a specific error, attaching errors screenshot. Can anyone help me about fixing this issue? 

Answer: The linear_assignment function is deprecated in 0.21 and will be removed from 0.23, but sklearn.utils.linear_assignment_ can be replaced by scipy.optimize.linear_sum_assignment. You can use: from scipy.optimize import linear_sum_assignment as linear_assignment  then you can run the file and don't need to change the code.