eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Am working on a binary classification problem with 1000 rows and 15 features. Currently am using Lime to explain the predictions of each instance. I use the below code to generate explanations for full test dataframe test_indx_list = X_test.index.tolist() test_dict={} for n in test_indx_list: exp = explainer.explain_instance(X_test.loc[n].values, model.predict_proba, num_features=5) a=exp.as_list() test_dict[n] = a  But this is not efficient. Is there any alternative approach to generate explanation/ get feature contributions quicker?

Answer: From what the docs show, there isn't currently an option to do batch explain_instance, although there are plans for it. This should help a lot with speed on newer versions later on. What seems to be the most appropriate change to get better speed is decreasing the number of samples used to learn the linear model. explainer.explain_instance(... num_features=5, num_samples=2500)  The default value for num_samples is 5000, which can be much more than you need depending on your model, and is currently the argument that will most affect the speed of the explainer. Another approach would be to try adding parallelization to the snippet. It's a more complex solution where you run multiple instances of the snippet at the same time, and gather the results at the end. For that, I leave a link, but really it's not something I can give a snippet right out of the box.
Question: So, I'm working on a small web application that has a small canvas, the user is supposed to draw something and then I want to do some python with the image from that canvas. Like this:  This is working fine. When I press "Click me!", I call a JS function that POST the image to my Flask server. And this is also working, the thing is that after I receive the image, I want to render a new page to show some results, but this page is not rendering at all. Now, I'm completely new to Javascript, barely know anything, basically every JS I'm using is copy pasted from the internet, so I'm assuming I must be missing something very simple, I just don't know what. Here is the server code: from flask import Flask, render_template, request import re from io import BytesIO import base64 from PIL import Image app = Flask(__name__) @app.route("/") def hello_world(): return render_template('drawing.html') @app.route("/hook", methods=['POST', 'GET']) def hook(): image_data = re.sub('^data:image/.+;base64,', '', request.form['imageBase64']) im = Image.open(BytesIO(base64.b64decode(image_data))) im.show() return render_template('results.html')  The first route just opens the canvas, the second is executed after the client's request, which has this function: function surprise() { var dataURL = canvas.toDataURL(); $.ajax({ type: "POST", url: "http://127.0.0.1:5000/hook", data:{ imageBase64: dataURL } }).done(function() { console.log('sent'); });}  Apparently, this is working. I have the line: im.show()  Just so I can see the image and I get exactly the drawing from the canvas, which I'm now supposed to work on:  but the results page is not rendered afterwards, nothing happens. Why?

Answer: The problem is that you are returning the page to view in response to the call that posts the image. Instead, you should return a response (for example in a json format) containing the information regarding the result of the call just made (i.e. the post of the image) and consequently, on the client side, you must redirect the user to the appropriate page . To understand why it is not working, print the content of the response returned from the call made var response = ''; $.ajax({ type: "POST", url: "http://127.0.0.1:5000/hook", data:{ imageBase64: dataURL }, success : function(text) { response = text; } }); alert(response);
Question: I'm having a strange problem with VSCode's python testing functionality. When I try to discover tests I get the following error: > conda run -n sandbox --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear . cwd: . [ERROR 2022-1-3 21:49:47.851]: Error discovering pytest tests: [r [Error]: EnvironmentLocationNotFound: Not a conda environment: /Users/david.hoffman/miniconda3/envs/sandbox/envs/sandbox  But obviously there's a duplication error: /Users/david.hoffman/miniconda3/envs/sandbox/envs/sandbox. If I run this command directly in the terminal I get the expected output and no errors: conda run -n sandbox --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear  I'm completely stumped as there doesn't seem to be any settings that would affect this. I tried reinstalling VSCode from scratch (after removing all the local files) same with conda.

Answer: Two ways I've found to fix:  Change the name of the conda environment. Just cloning sandbox to boxsand did the trick Add python.condaPath variable to VSCode's preferences
Question: I have the following dataframe where col2 is a dictionary with a list of tuples as values. The keys are consistantly 'added' and 'deleted' in the whole dataframe. Input df     col1 col2     value1 {'added': [(59, 'dep1_v2'), (60, 'dep2_v2')], 'deleted': [(59, 'dep1_v1'), (60, 'dep2_v1')]}   value 2 {'added': [(61, 'dep3_v2')], 'deleted': [(61, 'dep3_v1')]}     Here's a copy-pasteable example dataframe: jsons = ["{'added': [(59, 'dep1_v2'), (60, 'dep2_v2')], 'deleted': [(59, 'dep1_v1'), (60, 'dep2_v1')]}", "{'added': [(61, 'dep3_v2')], 'deleted': [(61, 'dep3_v1')]}"] df = pd.DataFrame({"col1": ["value1", "value2"], "col2": jsons})  edit col2 directly comes from the diff_parsed field of pydriller output I want to "explode" col2 so that I obtain the following result: Desired output     col1 number added deleted     value1 59 dep1_v2 dep1_v1   value1 60 dep2_v2 dep2_v1   value2 61 dep3_v2 dep3_v1     So far, I tried the following: df = df.join(pd.json_normalize(df.col2)) df.drop(columns=['col2'], inplace=True)  The above code is simplified. I first manipulate the column to convert to proper json. It was in an attempt to first explode on 'added' and 'deleted' and then try to play around with the format to obtain what I want...but the list of tuples is not preserved and I obtain the following:     col1 added deleted     value1 59, dep1_v2, 60, dep2_v2 59, dep1_v1, 60, dep2_v1   value2 61, dep3_v1 61, dep3_v2     Thanks

Answer: Well this certainly isn't elegant, but here's a potential solution that is at least easier to understand and reason about: def explode_records(df): new_records = [] def map_dict_to_row(value, col2_dict): temp = {} for number, added in col2_dict["added"]: temp[number] = {"value": value, "number": number, "added": added} for number, deleted in col2_dict["deleted"]: if number in temp: temp[number] = {**temp[number], "deleted": deleted} else: temp[number] = {"value": value, "deleted": deleted} new_records.extend(list(temp.values())) df.apply(lambda row: map_dict_to_row(row.col1, row.col2), axis=1) # assumes col2 is a dict return pd.DataFrame(new_records)  Usage: In [4]: explode_records(df) Out[4]: value number added deleted 0 value1 59 dep1_v2 dep1_v1 1 value1 60 dep2_v2 dep2_v1 2 value 2 61 dep3_v2 dep3_v1  Note that I got value 2 from your original data. I'm assuming it's just a typo, and not that you also need value x -> valuex functionality. I wasn't able to get the other solution working, so I wasn't able to compare its performance vs mine.
Question: I have a test that makes a thing, validates it, deletes the thing and confirms it was deleted. def test_thing(): thing = Thing() # Simplified, it actually takes many lines to make a thing assert thing.exists thing.delete() # Simplified, it also takes a few lines to delete it assert thing.deleted  Next I want to make many more tests that all use the thing, so it's a natural next step to move the thing creation/deletion into a fixture @pytest.fixture def thing(): thing = Thing() # Simplified, it actually takes many lines to make a thing yield thing thing.delete() # Simplified, it also takes a few lines to delete it def test_thing(thing): assert thing.exists def test_thing_again(thing): # Do more stuff with thing ...  But now I've lost my assert thing.deleted. I feel like I have a few options here, but none are satisfying.  I could assert in the fixture but AFAIK it's bad practice to put assertions in the fixture because if it fails it will result in an ERROR instead of a FAIL. I could keep my original test and also create the fixture, but this would result in a lot of duplicated code for creating/deleting the thing. I can't call the fixture directly because I get a Fixture called directly exception so I could move the thing creation out into a generator that is used by both the fixture and the test. This feels clunky though and what happens if my thing fixture needs to use another fixture?  What is my best option here? Is there something I haven't thought of?

Answer: If you want to test that a "thing" is deleted, make a fixture without teardown, delete it in the test, then assert if it is deleted. @pytest.fixture def thing_create(): # Perform all the creation steps thing = Thing() ... yield thing def thing_delete(thing): # Perform all the deletion steps ... thing.delete() @pytest.fixture def thing_all(thing_create): yield thing_create thing_delete(thing_create) def test_thing(thing_all): assert thing_all.exists def test_thing_again(thing_create): thing_delete(thing_create) assert thing_create.deleted
Question: I have two arrays, one is a list of values and one is a list of IDs corresponding to each value. Some IDs have multiple values. I want to create a new array that contains the maximum value recorded for each id, which will have a length equal to the number of unique ids. Example using a for loop: import numpy as np values = np.array([5, 3, 2, 6, 3, 4, 8, 2, 4, 8]) ids = np.array([0, 1, 3, 3, 3, 3, 5, 6, 6, 6]) uniq_ids = np.unique(ids) maximums = np.ones_like(uniq_ids) * np.nan for i, id in enumerate(uniq_ids): maximums[i] = np.max(values[np.where(ids == id)]) print(uniq_ids) print(maximums)  [0 1 3 5 6] [5. 3. 6. 8. 8.]  Is it possible to vectorize this so it runs fast? I'm imagining a one-liner that can create the "maximums" array using only NumPy functions, but I haven't been able to come up with anything that works.

Answer: np.lexsort sorts by multiple columns. However, this is not compulsory. You can sort ids first and then choose maximum item of each divided group using numpy.maximum.reduceat def mathfux(values, ids, return_groups=False): argidx = np.argsort(ids) #70% time ids_sort, values_sort = ids[argidx], values[argidx] #4% time div_points = np.r_[0, np.flatnonzero(np.diff(ids_sort)) + 1] #11% time (the most part for np.flatnonzero) if return_groups: return ids[div_points], np.maximum.reduceat(values_sort, div_points) else: return np.maximum.reduceat(values_sort, div_points) mathfux(values, ids, return_groups=True) >>> (array([0, 1, 3, 5, 6]), array([5, 3, 6, 8, 8])) mathfux(values, ids) >>> mathfux(values, ids) array([5, 3, 6, 8, 8])  Usually, some parts of numpy codes could be optimised further in numba. Note that np.argsort is a bottleneck in majority of groupby problems which can't be replaced by any other method. It is unlikely to be improved soon in numba or numpy. So you are reaching an optimal performance here and can't do much in further optimisations.
Question: I’m running a subprocess in full trace mode and displaying it using logger.info()  > std = subprocess.run(subprocess_cmd, shell=True, > universal_newlines=True, stdout=subprocess.PIPE, > stderr=subprocess.PIPE) > > all_stdout = all_stdout + std.stdout + ‘\n’ all_stderr = all_stderr + > std.stderr + ‘\n’ > > logger.info(‘\nstdout:\n’ + all_stdout + ‘\nstderr:\n’ + all_stderr)   But I’m getting the below error when the subprocess is getting printed. UnicodeDecodeError: ‘utf-8’ codec can’t decide byte 0x90 in position 4024984:invalid start byte I have tried giving universal_lines as False but it throws TypeError: must be str, not bytes. I have also tried this - std = subprocess.run(subprocess_cmd, shell=True, stdout=subprocess.PIPE,  > stderr=subprocess.PIPE, env=environ, encoding=‘utf-8’)   This gives me same UnicodeDecodeError.

Answer: Use the encoding encoding="unicode_escape" instead of encoding="utf-8"
Question: what is python kubernetes client equivalent for kubectl get deploy -o yaml  CRUD python Client example i referred this example for getting python deployment but there is no read deployment option

Answer: read_namespaced_deployment() does the thing: from kubernetes import client, config config.load_kube_config() api = client.AppsV1Api() deployment = api.read_namespaced_deployment(name='foo', namespace='bar')
Question: I currently have a connection string in the format of: "localhost://username:password@data_quality:5432"  What is the best method of using this to connect to my database using psycopg2? e.g.: connection = psycopg2.connect(connection_string) 

Answer: You could make use of urlparse, creating a dictionary that matches psycopg's connection arguments: import psycopg2 from urllib.parse import urlparse conStr = "postgres://username:password@localhost:5432/data_quality" p = urlparse(conStr) pg_connection_dict = { 'dbname': p.path[1:], 'user': p.username, 'password': p.password, 'port': p.port, 'host': p.hostname } print(pg_connection_dict) con = psycopg2.connect(**pg_connection_dict) print(con)  Out: {'dbname': 'data_quality', 'user': 'username', 'password': 'password', 'port': 5432, 'host': 'localhost'} <connection object at 0x105f58190; dsn: 'user=xxx password=xxx dbname=xxx host=xxx port=xxx', closed: 0>
Question: In Python there are multiple DateTime parsers which can parse a date string automatically without providing the datetime format. My problem is that I don't need to cast the datetime, I only need the datetime format. Example: From "2021-01-01", I want something like "%Y-%m-%d" or "yyyy-MM-dd". My only idea was to try casting with different formats and get the successful one, but I don't want to list every possible format. I'm working with pandas, so I can use methods that work either with series or the string DateTime parser. Any ideas?

Answer: In pandas, this is achieved by pandas.tseries.api.guess_datetime_format from pandas.tseries.api import guess_datetime_format guess_datetime_format('2021-01-01') # '%Y-%m-%d'  As there will always be an ambiguity on the day/month, you can specify the dayfirst case: guess_datetime_format('2021-01-01', dayfirst=True) # '%Y-%d-%m'
Question: I am new to Python development, I am writing test cases using pytest where I need to mock some behavior. Googling best mocking library for pytest, has only confused me. I have seen unittest.mock, mock, mocker and pytest-mock. Not really sure which one to use. Can someone please explain me the difference between them and also recommend me one?

Answer: pytest-mock is a thin wrapper around mock. mock is since python 3.3. actually the same as unittest.mock. I don't know if mocker is another library, I only know it as the name of the fixture provided by pytest-mock to get mocking done in your tests. I personally use pytest and pytest-mock for my tests, which allows you to write very concise tests like from pytest_mock import MockerFixture @pytest.fixture(autouse=True) def something_to_be_mocked_everywhere(mocker): mocker.patch() def tests_this(mocker: MockerFixture): mocker.patch ... a_mock = mocker.Mock() ... ...  But this is mainly due to using fixtures, which is already pointed out is what pytest-mock offers.
Question: def func(df_a: pd.DataFrame, df_b: pd.DataFrame) -> (pd.DataFrame, pd.DataFrame):  Pylance is advising to modify this line with two solution proposed. What would be the pros and cons of each one if there is any significant difference?  Tuple expression not allowed in type annotation Use Tuple[T1, ..., Tn] to indicate a tuple type or Union[T1, T2] to indicate a union type 

Answer: 2023 edit In newer versions of Python (>=3.10), you should use:  tuple[A, B, C] instead of Tuple[A, B, C] (yes, that's the built-in tuple function) A | B instead of Union[A, B]  The answer itself is still relevant, even if the newer style makes the difference between Tuple/tuple and Union/| more apparent. Original answer They mean different things:  Tuple[A, B, C] means that the function returns a three-element tuple with the A B C data types:  def f() -> Tuple[str, int, float]: return 'hello', 10, 3.33   Union[A, B] means that the function returns an object of either A or B data type:  import random def f() -> Union[str, int]: if random.random() > 0.5: return 'hello' else: return 10  In your case, it looks like you want to use Tuple[pd.DataFrame, pd.DataFrame].
Question: I need to install python 3.8.10 in a container running ubuntu 16.04. 16.04 has no support anymore, so I need a way to install it there manually.

Answer: This follows from here Add the following to your dockerfile, and change the python version as needed. When the docker is up, python3.8 will be available in /usr/local/bin/python3.8 # compile python from source - avoid unsupported library problems RUN apt update -y && sudo apt upgrade -y && \ apt-get install -y wget build-essential checkinstall libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev && \ cd /usr/src && \ sudo wget https://www.python.org/ftp/python/3.8.10/Python-3.8.10.tgz && \ sudo tar xzf Python-3.8.10.tgz && \ cd Python-3.8.10 && \ sudo ./configure --enable-optimizations && \ sudo make altinstall  Please note the following (standard [and quicker] way of installing) does not work for old ubuntu versions, due to end of support RUN apt-get update && \ apt-get install -y software-properties-common && \ add-apt-repository -y ppa:deadsnakes/ppa && \ apt-get update && \ apt install -y python3.8   See also this to install into /usr/bin
Question: I am new to FastAPI and I've been having this problem with importing my other files. I get the error:‍‍‍‍ from . import schemas ImportError: attempted relative import with no known parent package  For context, the file I am importing from is a Folder called Blog. I saw certain StackOverflow answers saying that instead of from . import schemas I should write from Blog import schemas. And even though their solution is right and I don't get any errors while running the python program, When I try running FastAPI using uvicorn, I get this error and my localhost page doesn't load.  File "./main.py", line 2, in <module> from Blog import schemas ModuleNotFoundError: No module named 'Blog'  The file structure looks like this:  The code to the main file looks like this: from fastapi import FastAPI from Blog import schemas, models from database import engine app = FastAPI() models.Base.metadata.create_all(engine) @app.post('/blog') def create(request: schemas.Blog): return request  schemas.py from pydantic import BaseModel class Blog(BaseModel): title: str body: str  database.py from sqlalchemy import create_engine from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker SQLALCHAMY_DATABASE_URL = 'sqlite:///./blog.db' engine = create_engine(SQLALCHAMY_DATABASE_URL, connect_args={"check_same_thread": False}) SessionLocal = sessionmaker(bind=engine, autocommit=False, autoflush=False) Base = declarative_base()  models.py from sqlalchemy import * from database import Base class Blog(Base): __tablename__ = 'blogs' id = Column(Integer, primary_key=True, index=True) title = Column(String) body = Column(String)  The SwaggerUI is not loading either. Any help would be greatly appreciated! :)

Answer: Since your schemas.py and models.py files are in the same directory as your main.py file, you should import those two modules as follows, instead of using from Blog import schemas, models: import schemas, models  For more details and examples please have a look at this answer, as well as this answer and this answer.
Question: I am new to python and Iam trying to manipulate some data but it keeps showing me this erro message  UserWarning: Parsing '13/01/2021' in DD/MM/YYYY format. Provide format or specify infer_datetime_format=True for consistent parsing. cache_array = _maybe_cache(arg, format, cache, convert_listlike)  This is my code import pandas as pd import matplotlib.pyplot as plt dataLake = pd.read_csv("datalake - Data lake.csv", parse_dates=["Day"]) dataLake = dataLake.rename(columns={"Day":"day"}) dataLake = dataLake.rename(columns={"Agent":"agent"}) dataLake["day"] = pd.to_datetime(dataLake.day) print(dataLake.head()) 

Answer: In your case you need to set the dayfirst param to true, like this: pd.to_datetime(dataLake.day, dayfirst=True)  or you can set a format (but you don't need to in your case), like this: pd.to_datetime(dataLake.day, format="%d/%m/%y")
Question: I'm getting the following error:  File "/home/ron/rzg2l_bsp_v1.3/poky/bitbake/lib/bb/compat.py", line 7, in <module> from collections import MutableMapping, KeysView, ValuesView, ItemsView, OrderedDict ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)  and Googling revealed that flask has to be >=2.0, so I did $ sudo pacman -Syu python-flask  which installed version (2.0.2-3) which did not resolve the issue. Further searching revealed that babelfish needs to be upgraded too, so I did: $ python3.10 -m pip install babelfish -U  which showed me: Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: babelfish in /home/ron/.local/lib/python3.10/site-packages (0.6.0) Collecting babelfish Using cached babelfish-0.6.0-py3-none-any.whl (93 kB) Downloading babelfish-0.5.5.tar.gz (90 kB) |████████████████████████████████| 90 kB 406 kB/s  but I'm still getting the same error. Can anyone tell what else I'm missing?

Answer: You need to import collections.abc Here the link to doc >>> from collections import MutableMapping Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py) >>> from collections.abc import MutableMapping   Deprecated since version 3.3, will be removed in version 3.10: Moved Collections Abstract Base Classes to the collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.9. Ref. https://docs.python.org/3.9/library/collections.html
Question: How do I sort an Arrow table in PyArrow? There does not appear to be a single function that will do this, the closest is sort_indices.

Answer: PyArrow includes Table.sort_by since 7.0.0, no need to manually call the compute functions (reference) table = pa.table([ pa.array(["a", "a", "b", "b", "b", "c", "d", "d", "e", "c"]), pa.array([15, 20, 3, 4, 5, 6, 10, 1, 14, 123]), ], names=["keys", "values"]) sorted_table = table.sort_by([("values", "ascending")])
Question: I have recently started using VS code for Python development. I am unable to figure out how to launch an interactive terminal while debugging, with the program state loaded-in . For example, consider the following code, import numpy as np A = np.array([1, 2, 3]) B = np.zeros() C = A/B \\ <--- Breakpoint here  I want to set a breakpoint at C = A/B and as soon as the breakpoint hit, I want to launch an interactive terminal that holds the state of my program. So that I can play around with variables in the terminal. This is simple and straightforward in other Python IDEs like Spyder and Pycharm. How do I do this with VS Code?

Answer: There's the Python debugging console in VSCode. When your code stops on a breakpoint, you can click on the debug console button to open an interactive Python console with your current program state loaded in.