eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a large dataset (~5 Mio rows) with results from a Machine Learning training. Now I want to check to see if the results hit the "target range" or not. Lets say this range contains all values between -0.25 and +0.25. If it's inside this range, it's a Hit, if it's below Low and on the other side High. I now would create this three columns Hit, Low, High and calculate for each row which condition applies and put a 1 into this col, the other two would become 0. After that I would group the values and sum them up. But I suspect there must be a better and faster way, such as calculate it directly while grouping.  Data import pandas as pd df = pd.DataFrame({"Type":["RF", "RF", "RF", "MLP", "MLP", "MLP"], "Value":[-1.5,-0.1,1.7,0.2,-0.7,-0.6]}) +----+--------+---------+ | | Type | Value | |----+--------+---------| | 0 | RF | -1.5 | <- Low | 1 | RF | -0.1 | <- Hit | 2 | RF | 1.7 | <- High | 3 | MLP | 0.2 | <- Hit | 4 | MLP | -0.7 | <- Low | 5 | MLP | -0.6 | <- Low +----+--------+---------+   Expected Output pd.DataFrame({"Type":["RF", "MLP"], "Low":[1,2], "Hit":[1,1], "High":[1,0]}) +----+--------+-------+-------+--------+ | | Type | Low | Hit | High | |----+--------+-------+-------+--------| | 0 | RF | 1 | 1 | 1 | | 1 | MLP | 2 | 1 | 0 | +----+--------+-------+-------+--------+ 

Answer: You could use cut to define the groups and pivot_table to reshape: (df.assign(group=pd.cut(df['Value'], [float('-inf'), -0.25, 0.25, float('inf')], labels=['Low', 'Hit', 'High'])) .pivot_table(index='Type', columns='group', values='Value', aggfunc='count') .reset_index() .rename_axis(None, axis=1) )  Or crosstab: (pd.crosstab(df['Type'], pd.cut(df['Value'], [float('-inf'), -0.25, 0.25, float('inf')], labels=['Low', 'Hit', 'High']) ) .reset_index().rename_axis(None, axis=1) )  output:  Type Low Hit High 0 MLP 2 1 0 1 RF 1 1 1
Question: I am trying to save a simple plotly figure to a directory. I understand it needs kaleido (I have version '0.2.1') and also at least plotly '5.3.1' which are installed. However trying to save the image I get the following error: fig.write_image(path) ValueError: Image export using the "kaleido" engine requires the kaleido package, which can be installed using pip: $ pip install -U kaleido  Why is this occuring when all the required packages are there?

Answer: I've got exactly the same issue on google colab. Even after installation kaleido, the same error. Accidentally I've found fix, it seems necesarry to import kaleido first: !pip install kaleido import kaleido #required kaleido.__version__ #0.2.1 import plotly plotly.__version__ #5.5.0 #now this works: import plotly.graph_objects as go fig = go.Figure() fig.write_image('aaa.png')
Question: Setup Let's assume the following undirected graph: import networkx as nx G = nx.from_edgelist([(0, 3), (0, 1), (2, 5), (0, 3)]) G.add_nodes_from(range(7))   or even adding the (1, 3) edge (it doesn't matter here):  The connected components are: list(nx.connected_components(G)) # [{0, 1, 3}, {2, 5}, {4}, {6}]  Question Is it possible to generate the graph G from the list of connected components directly with networkx? Or using a simple method? The only solution I found so far is to generate the successive edges or all combinations of nodes per group and feed it to nx.from_edgelist, then to add the single nodes with add_nodes_from: from itertools import pairwise, chain l = [{0, 1, 3}, {2, 5}, {4}, {6}] G = nx.from_edgelist(chain.from_iterable(pairwise(e) for e in l)) G.add_nodes_from(set.union(*l))  or for all edges: from itertools import combinations, chain l = [{0, 1, 3}, {2, 5}, {4}, {6}] G = nx.from_edgelist(chain.from_iterable(combinations(e, 2) for e in l)) G.add_nodes_from(set.union(*l)) 

Answer: An alternative to itertools.pairwise is networkx.path_graph. An alternative to itertools.combinations is networkx.complete_graph. These two networkx functions return a new graph, not a list of edges, so you can combine them with networkx.compose_all. Note also union_all and disjoint_union_all as alternatives to compose_all. import networkx as nx l = [{0, 1, 3}, {2, 5}, {4}, {6}] G = nx.compose_all(map(nx.path_graph, l)) H = nx.compose_all(map(nx.complete_graph, l)) print(G.nodes, G.edges) # [0, 1, 3, 2, 5, 4, 6] [(0, 1), (1, 3), (2, 5)] print(H.nodes, H.edges) # [0, 1, 3, 2, 5, 4, 6] [(0, 1), (0, 3), (1, 3), (2, 5)]  I haven't actually run benchmarks, but I suspect creating several graphs and composing them might be slower than creating lists of edges and chaining them to create only one graph.
Question: My python interpreter version does not show up at the bottom of the status bar on VS code on my Mac, it used to but suddenly stopped. Everything works but it just doesn’t show, I tried many possible solutions such as:  right clicking the bar to have the Python Extension checked (which I don’t even have an option to check) uninstalling all the extensions then reinstalling it but it didn’t help. Even after restarting my computer. I also can't seem to add python.pythonPath in my settings.json file if that has something to do with it and if it does how can I get that? When I try to add that in my VS code settings.json, it says 'unknown configuration"  Basically I would just like to see the python version on the status bar. status bar on vs code

Answer: Turned out it was placed to a new place in the status bar. Here's how to pin it on the status bar now:  Hover over the {} next to the Python language chooser Click the pin icon The selection of the Python environment becomes pinned to the status bar on the right hand side
Question: Unable to import functions from scipy module. Gives error : from scipy.signal.signaltools import _centered Cannot import name '_centered' from 'scipy.signal.signaltools' scipy.__version__ 1.8.0 

Answer: If you need to use that specific version of statsmodels 0.12.x with scipy 1.8.0 I have the following hack. Basically it just re-publishes the existing (but private) _centered function as a public attribute to the module already imported in RAM. It is a workaround, and if you can simply upgrade your dependencies to the latest versions. Only use this if you are forced to use those specific versions. import scipy.signal.signaltools def _centered(arr, newsize): # Return the center newsize portion of the array. newsize = np.asarray(newsize) currsize = np.array(arr.shape) startind = (currsize - newsize) // 2 endind = startind + newsize myslice = [slice(startind[k], endind[k]) for k in range(len(endind))] return arr[tuple(myslice)] scipy.signal.signaltools._centered = _centered
Question: I know that several similar questions exist on this topic, but to my knowledge all of them concern an async code (wrongly) written by the user, while in my case it comes from a Python package. I have a Jupyter notebook whose first cell is ! pip install numpy ! pip install pandas  and I want to automatically play the notebook using Papermill. No problem on my local machine (Windows 11 with Python 3.7): I install iPyKernel and Papermill and everything is fine. The problem is when I try to do the same on my BitBucket pipeline (Python image 3-alpine, but it happens under different others); the first cell throws the following error: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 461, in dispatch_queue await self.process_one() File "/usr/local/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 450, in process_one await dispatch(*args) TypeError: object NoneType can't be used in 'await' expression  that makes the script stop at the 2nd cell, where I import numpy. If it can be relevant, I've "papermilled" under the GitLab CI without any problem in the past.

Answer: Seems to be a bug in ipykernel 6.9.0 - options that worked for me:  upgrade to 6.9.1 (latest version as of 2022-02-22); e.g. via pip install ipykernel --upgrade downgrade to 6.8.0 (if upgrading messes with other dependencies you might have); e.g. via pip install ipykernel==6.8.0
Question: I've been using a python interpreter that I set in my venv the whole time for this project. Recently I changed my python interpreter which I've set as a default interpreter in my user settings like the vs docs vs code python environments describes it and also set in my JSON settings file refuses to load instead. I'm getting the default conda python interpreter with no other options. here is my Json file: settings.json { "python.defaultInterpreterPath": "C:/Users/houst/Envs/Mach/Scripts/python.exe" } 

Answer: I think you should use the \ instead of the / in the path. For safety just copy paste the path from the windows explorer.  Another Solution : Writing the path in Json should definitely work. But I prefer using the functionality vscode provides. Now come to my solution. press ctrl+shift+p to open the search bar. now type >Python:Select Interpreter. This should allow you to select the existing interpreter or to add the interpreter path in the vscode. I'm attaching a screen shot for your convenient.  You should keep in mind that, if you add the Python interpreted path in environment path of your system, vscode should automatically find the interpreter.
Question: My app.py file from flask import Flask from flask_sqlalchemy import SQLAlchemy from flask import Flask from flask_sqlalchemy import SQLAlchemy app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres:////tmp/test.db' db = SQLAlchemy(app) # refer https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/#flask_sqlalchemy.SQLAlchemy  One of my model classes, where I imported db from app import db Base = declarative_base() # User class class User(db.Model, Base): id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String(80), unique=True, nullable=False) email = db.Column(db.String(120), unique=True, nullable=False) def __repr__(self): return '<User %r>' % self.username def get_user_by_id(self, id): return self.query.get(id)  My database has the same set of tables in different schema (multi-tenancy) and there I need to select the schema as per the request initiated by a particular tenant on the fly by using before_request (grabbing tenant_id from subdomain URL). I found Postgres provides selecting the schema name on fly by using schema_translate_map ref. https://docs.sqlalchemy.org/en/14/core/connections.html#translation-of-schema-names and that is under execution_options https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Connection.execution_options In my above code snippet where you see db = SQLAlchemy(app), as per official documentation, two parameters can be set in SQLAlchemy objct creation and they are - session_options and engine_options, but no execution_options ref. https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/#flask_sqlalchemy.SQLAlchemy But how do I set schema_translate_map setting when I am creating an object of SQLAlchemy I tried this - db = SQLAlchemy(app, session_options={ "autocommit": True, "autoflush": False, "schema_translate_map": { None: "public" } } )  But obviously, it did not work, because schema_translate_map is under execution_options as mentioned here https://docs.sqlalchemy.org/en/14/core/connections.html#translation-of-schema-names Anyone has an idea, how to set schema_translate_map at the time of creating SQLAlchemy object. My goal is to set it dynamically for each request. I want to control it from this centralized place, rather than going in each model file and specifying it when I execute queries. I am aware of doing this differently as suggested here https://stackoverflow.com/a/56490246/1560470 but my need is to set somewhere around db = SQLAlchemy(app) in app.py file only. Then after I import db in all my model classes (as shown above) and in those model classes, all queries execute under the selected schema.

Answer: I found a way to accomplish it. This is what needed db = SQLAlchemy(app, session_options={ "autocommit": True, "autoflush": False }, engine_options={ "execution_options": { "schema_translate_map": { None: "public", "abc": "xyz" } } } )
Question: I am implementing a simple chatbot using keras and WebSockets. I now have a model that can make a prediction about the user input and send the according answer. When I do it through command line it works fine, however when I try to send the answer through my WebSocket, the WebSocket doesn't even start anymore. Here is my working WebSocket code: @sock.route('/api') def echo(sock): while True: # get user input from browser user_input = sock.receive() # print user input on console print(user_input) # read answer from console response = input() # send response to browser sock.send(response)  Here is my code to communicate with the keras model on command line: while True: question = input("") ints = predict(question) answer = response(ints, json_data) print(answer)  Used methods are those: def predict(sentence): bag_of_words = convert_sentence_in_bag_of_words(sentence) # pass bag as list and get index 0 prediction = model.predict(np.array([bag_of_words]))[0] ERROR_THRESHOLD = 0.25 accepted_results = [[tag, probability] for tag, probability in enumerate(prediction) if probability > ERROR_THRESHOLD] accepted_results.sort(key=lambda x: x[1], reverse=True) output = [] for accepted_result in accepted_results: output.append({'intent': classes[accepted_result[0]], 'probability': str(accepted_result[1])}) print(output) return output def response(intents, json): tag = intents[0]['intent'] intents_as_list = json['intents'] for i in intents_as_list: if i['tag'] == tag: res = random.choice(i['responses']) break return res  So when I start the WebSocket with the working code I get this output:  * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) * Restarting with stat * Serving Flask app 'server' (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: on  But as soon as I have anything of my model in the server.py class I get this output: 2022-02-13 11:31:38.887640: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2022-02-13 11:31:38.887734: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) Metal device set to: Apple M1 systemMemory: 16.00 GB maxCacheSize: 5.33 GB  It is enough when I just have an import at the top like this: from chatty import response, predict - even though they are unused.

Answer: I am devastated, I just wasted 2 days into the dumbest possible issue (and fix) I still had the while True: question = input("") ints = predict(question) answer = response(ints, json_data) print(answer)  in my model file, so the server didn't start. The fix was to delete it and now it works fine.
Question: def get_latest_file_movement(**kwargs): get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s')) s3 = boto3.client('s3') objs = s3.list_objects_v2(Bucket='my-bucket',Prefix='prefix')['Contents'] last_added = [obj['Key'] for obj in sorted(objs, key=get_last_modified, reverse=True)][0] return last_added  Above code gets me the latest file however i only want the files ending with 'csv'

Answer: You can check if they end with .csv: def get_latest_file_movement(**kwargs): get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s')) s3 = boto3.client('s3') objs = s3.list_objects_v2(Bucket='my-bucket',Prefix='prefix')['Contents'] last_added = [obj['Key'] for obj in sorted(objs, key=get_last_modified, reverse=True) if obj['Key'].endswith('.csv')][0] return last_added
Question: I am trying to run an image-based project on colab. I found the project on github. Everything runs fine till I reached the cell with the following code: import keras from keras.preprocessing.image import ImageDataGenerator from keras.applications.resnet50 import preprocess_input, ResNet50 from keras.models import Model from keras.layers import Dense, MaxPool2D, Conv2D  When I run it, the following output is observed: --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-24-173cbce466d6> in <module>() 1 import keras 2 from keras.preprocessing.image import ImageDataGenerator ----> 3 from keras.applications.resnet50 import preprocess_input, ResNet50 4 from keras.models import Model 5 from keras.layers import Dense, MaxPool2D, Conv2D ModuleNotFoundError: No module named 'keras.applications.resnet50' ---------------------------------------------------------------------------  It's running 2.7.0 keras, connected to a TPU runtime. I tried !pip installing the said module but no use. I even tried running a demo resnet50 project too but got the same error. Can anyone please help me solve the error?

Answer: from tensorflow.keras.applications.resnet50 import ResNet50
Question: when i am trying to debug my code the debugger stops at line 2 and doenst respond to any commands (like go to next line). I am using python 3.9.7. This is what the console looks like:  If I try to stop the debugger this happens:  The only thing I can do then is close the console.

Answer: I had recently the same problem (using Python 3.8) and the solution was to revert an recent upgrade of qtconsole from 5.1.1 to 5.2.2. In case you use conda, the command to revert would be "conda install qtconsole=5.1.1".
Question: If a function can return None, shouldn't the type annotation use NoneType? For example, shouldn't we use this: from types import NoneType def my_function(num: int) -> int | NoneType: if num > 0: return num return None  instead of: def my_function(num: int) -> int | None: if num > 0: return num return None  ?

Answer: No. types.NoneType was removed in Python 3. Attempting to import NoneType from types will produce an ImportError in Python 3, before Python 3.10. (For Python 3.10, types.NoneType was reintroduced; however, for the purposes of type hinting, types.NoneType and None are equivalent, and you should prefer the latter for conciseness.) In Python 3.10, int | None is the way to describe a return type that could possibly be None. However, for Python versions earlier than 3.10, this syntax is unsupported, so you should use Optional[int] instead.
Question: Something just doesn't click internally for me with pythons logging despite reading the documentation. I have this code import logging logging.basicConfig(level=logging.INFO,format='%(levelname)s::%(message)s') LOG = logging.getLogger("__name__") LOG.info("hey")  If I run it from bash I get this: INFO::hey  If I run it in an aws lambda the "hey" doesn't shows up at all in the logs. I then did a test setting the level on the logger by adding this: LOG.setLevel(logging.INFO)  Run from bash I get the same thing I got before (desired format), but run from the lambda this shows up in the logs: [INFO] 2022-02-14T23:30:43.39Z eb94600a-af45-4124-99b6-d9651d6a3cf6 hey  Okay... that is pretty odd. The format is not the same as from bash. I thought I could rationalize the first example because the output on bash is actually going to stderr. And I then assumed the aws lamdba logs just don't grab that. But the second example is also going to stderr on bash, yet it shows up in the lambda logs but with the wrong format. So clearly I am missing something. What is going on under the hood here?

Answer: When your Lambda runs, a harness is running that does some basic bootstrap and then loads your module and invokes it. Part of that bootstrap in the AWS Lambda Python Runtime replaces the standard Python logger with its own:  logger_handler = LambdaLoggerHandler(log_sink) logger_handler.setFormatter( logging.Formatter( "[%(levelname)s]\t%(asctime)s.%(msecs)dZ\t%(aws_request_id)s\t%(message)s\n", "%Y-%m-%dT%H:%M:%S", ) ) logger_handler.addFilter(LambdaLoggerFilter())  This behavior is formally documented by AWS as AWS Lambda function logging in Python , under the section "Logging library".
Question: This just suddenly started happening where python pylint will highlight the whole function with blue squiggly lines when for a missing function docstring warning. How can I get it to only highlight the function definition or make a small indicator on the definition line. Its super annoying to get the whole file highlighted when you're developing. Here's an example of a missing class docstring. The whole file becomes ridiculous to work with. example of annoying behavior  Desired behaviour is just a small quiggle at the beginning of the line. Thats how it used to be. enter image description here

Answer: A solution for this is being actively discussed and developed at the Pylint project. The workarounds until a fix is merged are either to use an earlier version of VS Code (before January 2022) or Pylint (below 2.12.2). If the latter is desired, you can download a local copy and specify a custom path to Pylint in the Python extension settings.
Question: Here is my file structure and requirements.txt:  Getting ModuleNotFoundError, any help will be appreciated. main.py from fastapi import FastAPI from .import models from .database import engine from .routers import ratings models.Base.metadata.create_all(bind=engine) app = FastAPI() app.include_router(ratings.router) 

Answer: The error comes from the fact that you were not using the right environment and python version on VSCODE. Your environment knew your different packages, but VSCode probably did not take them into account. The solution was, in VSCODE: CTRL + SHIFT + P then Python:select interpreter and choose the version of python linked to your environment. You can try to change the version of python to see the consequences on your imports
Question: How do I use the constructor of a python Generic typed class? T = typing.TypeVar('T') class MyClass(typing.Generic[T]): def __init__(self, initialValue: typing.Iterable): self.values: T = T(initialValue) test = MyClass[tuple[int]]([1, 2, 3])  In this case I am expecting T(initialValue) to be equivalent to tuple(initialValue) but instead I get an error. "Exception has occurred: TypeError 'TypeVar' object is not callable" I guess that's not too surprising since that's not what typing was built for, but is there a workaround to accomplish this?

Answer: You'll need to take an explicit factory method. Type annotations only exist for compile-time purposes, and at runtime that T is just a TypeVar. Consider class MyClass(Generic[T]): def __init__(self, initialValue: Iterable[int], factory: Callable[[Iterable[int]], T]): self.values: T = factory(initialValue)  Then call it as test = MyClass([1, 2, 3], lambda x: tuple(x))  Note: It would be nice to just pass tuple as the second argument, but mypy seems to choke when converting that typename to a Callable. Other type checkers may be able to handle it; your mileage may vary.
Question: I am trying to build a package of .pyi stub files, for use with type annotations. I have this structure: m/ __init__.pyi sub.pyi  This works: >>> import m  This does not (including in Python 3.10): >>> import m.sub ModuleNotFoundError: No module named 'm.sub'  If I rename sub.pyi to sub.py, then I can import m.sub. What am I misunderstanding about where I can use .pyi files, or whether I can make multi-file stubs package?

Answer: cf PEP 484 -- Type Hints :  While stub files are syntactically valid Python modules, they use the .pyi extension to make it possible to maintain stub files in the same directory as the corresponding real module. This also reinforces the notion that no runtime behavior should be expected of stub files.  The .pyi files are not meant to be used by the Python interpreter itself but by type-checking tools. By default, a Python loader will not look for .pyi files, so you can't import them. But you can import a package (a directory whose name matches what you are importing). That's the reason of your error. Looking at the stubs files that PyCharm includes, the only imports it does is to actual .py files, never to .pyi files.