eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: my question is how to make a normal distribution graph from data frame in Python. I can find many information to make such a graph from random numbers, but I don't know how to make it from data frame. First, I generated random numbers and made a data frame. import numpy as np import pandas from pandas import DataFrame cv1 = np.random.normal(50, 3, 1000) source = {"Genotype": ["CV1"]*1000, "AGW": cv1} Cultivar_1=DataFrame(source)   Then, I tried to make a normal distribution graph. sns.kdeplot(data = Cultivar_1['AGW']) plt.xlim([30,70]) plt.xlabel("Grain weight (mg)", size=12) plt.ylabel("Frequency", size=12) plt.grid(True, alpha=0.3, linestyle="--") plt.show()   However, this is a density graph, not a normal distribution graph which is calculated using mean and standard deviation. Could you let me know which codes I need to use to make a normal distribution graph? Thanks!!

Answer: I found one solution to make a normal distribution graph from data frame. #Library import numpy as np import pandas as pd import matplotlib.pyplot as plt import scipy.stats as stats #Generating data frame x = np.random.normal(50, 3, 1000) source = {"Genotype": ["CV1"]*1000, "AGW": x} df = pd.DataFrame(source) # Calculating mean and Stdev of AGW df_mean = np.mean(df["AGW"]) df_std = np.std(df["AGW"]) # Calculating probability density function (PDF) pdf = stats.norm.pdf(df["AGW"].sort_values(), df_mean, df_std) # Drawing a graph plt.plot(df["AGW"].sort_values(), pdf) plt.xlim([30,70]) plt.xlabel("Grain weight (mg)", size=12) plt.ylabel("Frequency", size=12) plt.grid(True, alpha=0.3, linestyle="--") plt.show()
Question: I have a string with data that looks like this: str1 = "[2.4],[5],[2.54],[4],[3.36],[4.46],[3.36],[4],[3.63],[4.86],[4],[4.63]"  I would want to replace every second iteration of "],[" with "," so it will look like this: str2 = "[2.4,5],[2.54,4],[3.36,4.46],[3.36,4],[3.63,4.86],[4,4.63]"  Here is was I have so far: str1 = "[2.4],[5],[2.54],[4],[3.36],[4.46],[3.36],[4],[3.63],[4.86],[4],[4.63]" s2 = re.sub(r"],\[", ',', str1) print(s2)  I was trying to mess around with this: (.*?],\[){2}  But it does not seem to yield me the desired results. I tried using loops but I only managed to replace only the second occurrence and nothing after using this sample code I found here. And the code is: import re def replacenth(string, sub, wanted, n): where = [m.start() for m in re.finditer(sub, string)][n-1] before = string[:where] after = string[where:] after = after.replace(sub, wanted, 1) newString = before + after print(newString) For these variables: string = 'ababababababababab' sub = 'ab' wanted = 'CD' n = 5  Thank you.

Answer: You can use import re from itertools import count str1 = "[2.4],[5],[2.54],[4],[3.36],[4.46],[3.36],[4],[3.63],[4.86],[4],[4.63]" c = count(0) print( re.sub(r"],\[", lambda x: "," if next(c) % 2 == 0 else x.group(), str1) ) # => [2.4,5],[2.54,4],[3.36,4.46],[3.36,4],[3.63,4.86],[4,4.63]  See the Python demo. The regex is the same, ],\[, it matches a literal ],[ text. The c = count(0) initializes the counter whose value is incremented upon each match inside a lambda expression used as the replacement argument. When the counter is even, the match is replaced with a comma, else, it is kept as is.
Question: This is a simple question so I'm surprised that I can't find it asked on SO (apologies if I've missed it), and it always pops into my mind as I contemplate a refactor to replace a tuple by a NamedTuple. Can I unpack a typing.NamedTuple as arguments or as a destructuring assignment, like I can with a tuple?

Answer: Yes you certainly can. from typing import NamedTuple class Test(NamedTuple): a: int b: int t = Test(1, 2) # destructuring assignment a, b = t # a = 1 # b = 2 def f(a, b): return f"{a}{b}" # unpack f(*t) # '12'  Unpacking order is the order of the fields in the definition.
Question: I've defined two simple Python functions that take a single argument, raise an exception, and handle the raised exception. One function uses a variable to refer to the exception before raising/handling, the other does not: def refcount_unchanged(x): try: raise Exception() except: pass def refcount_increases(x): e = Exception() try: raise e except: pass  One of the resulting functions increases pythons refcount for its input argument, the other does not: import sys a = [] print(sys.getrefcount(a)) for i in range(3): refcount_unchanged(a) print(sys.getrefcount(a)) # prints: 2, 2, 2, 2 b = [] print(sys.getrefcount(b)) for i in range(3): refcount_increases(b) print(sys.getrefcount(b)) # prints: 2, 3, 4, 5  Can anyone explain why this happens?

Answer: It is a side effect of the "exception -> traceback -> stack frame -> exception" reference cycle from the __traceback__ attribute on exception instances introduced in PEP-344 (Python 2.5), and resolved in cases like refcount_unchanged in PEP-3110 (Python 3.0). In refcount_increases, the reference cycle can be observed by printing this: except: print(e.__traceback__.tb_frame.f_locals) # {'x': [], 'e': Exception()}  which shows that x is also referenced in the frame's locals. The reference cycle is resolved when the garbage collector runs, or if gc.collect() is called. In refcount_unchanged, as per PEP-3110's Semantic Changes, Python 3 generates additional bytecode to delete the target, thus eliminating the reference cycle: def refcount_unchanged(x): try: raise Exception() except: pass  gets translated to something like: def refcount_unchanged(x): try: raise Exception() except Exception as e: try: pass finally: e = None del e  Resolving the reference cycle in refcount_increases While not necessary (since the garbage collector will do its job), you can do something similar in refcount_increases by manually deleting the variable reference: def refcount_increases(x): e = Exception() try: raise e except: pass finally: # + del e # +  Alternatively, you can overwrite the variable reference and let the implicit deletion work: def refcount_increases(x): e = Exception() try: raise e # except: # - except Exception as e: # + pass  A little more about the reference cycle The exception e and other local variables are actually referenced directly by e.__traceback__.tb_frame, presumably in C code. This can be observed by printing this: print(sys.getrefcount(b)) print(gc.get_referrers(b)[0]) # <frame at ...>  Accessing e.__traceback__.tb_frame.f_locals creates a dictionary cached on the frame (another reference cycle) and thwarts the proactive resolutions above. print(sys.getrefcount(b)) print(gc.get_referrers(b)[0]) # {'x': [], 'e': Exception()}  However, this reference cycle will also be handled by the garbage collector.
Question: I have been watching several videos on Multiprocessing map function. I know that I can send one list as an argument to the function I want to target with Multiprocessing, and that will call the same function n times (dependent upon the size of that passed list). What I am struggling to do is what if I want to pass multiple arguments to that function? I basically have a List whose size is n (it can vary but for current case, its 209) My function requires 3 arguments...  the index of the list (0, 1, 2 etc.) Another list containing data A fixed integer value  I could have used the 2nd and 3rd arguments as global variables, but that doesn't work for me because I have to call the map function in a while loop... and in every another iteration, the values of these two will change. My function returns two values which I need to access in the function from where it was called. This is what I have tried but it didn't work for me, def main_fun(): with concurrent.futures.ProcessPoolExecutor() as executor: results = executor.map(MyFun, (row, pop[0].data, fitness) for row in range(0, len(pop[0].data))) for result in results: print(result)  I also tried to use ZIP function but again, with no success.

Answer: If your second and third arguments to your worker function (i.e. the first argument to map), then you can use method functools.partial to have the second and third arguments specified without resorting to the use of global variables. If your worker functions is, for example, foo, then: from concurrent.futures import ProcessPoolExecutor from functools import partial def foo(idx: int, lst: list, int_value: int): ... def main(): with ProcessPoolExecutor() as executor: worker = partial(foo, lst=pop[0].data, int_value=fitness) executor.map(worker, range(0, len(pop[0].data))) if __name__ == '__main__': main()  So now we only have to pass to map function worker, which will be called two fixed arguments, and a single iterable argument. If you are executing the map call in a loop, you will, of course, create a new worker functions by passing to functools.partial new arguments. For example: from concurrent.futures import ProcessPoolExecutor from functools import partial def foo(idx: int, lst: list, int_value: int): print(idx, lst[idx] * int_value, flush=True) def main(): l = [3, 5, 7] fitness = 9 with ProcessPoolExecutor() as executor: worker = partial(foo, lst=l, int_value=fitness) executor.map(worker, range(0, len(l))) if __name__ == '__main__': main()  Prints: 0 27 1 45 2 63
Question: I want to install gensim python package via pip install gensim But this error occurs and I have no idea what should I do to solve it.  running build_ext building 'gensim.models.word2vec_inner' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> gensim note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. 

Answer: If you fail to install plugins, you can download it from other repositories like this one: repository depends on the version of python and the system. for example: for windows 11(x64) and python 3.10 you should take this file: gensim‑4.1.2‑cp310‑cp310‑win_amd64.whl
Question: Tkinter background appears black upon running script no matter how I attribute the background colour. I'm using PyCharm CE 2021.3.2 on macOS 12.2.1. Python Interpreter = Python 3.8 with 5 packages (as follows):  Pillow 9.0.1 future 0.18.2 pip 22.0.3 setuptools 57.0.0 wheel 0.36.2  Window looks like this: Black, blank Tkinter window I've tried: import tkinter as tk window = tk.Tk() window.title("Test") window.geometry("600x400") window.mainloop()  Tried changing with window.configure(bg="white") as well as window['bg'] = "white" and window['background'] = "white" to no avail.

Answer: Thanks to @typedecker Issue was with Python 3.8 and the Monterey update. Fix: First install Python 3.10 then follow this tutorial: Creating Python 3.10 Virtual Env Then simply select the newly created virtual env in PyCharms and run.
Question: I am trying to call a local ES instance running on docker. I used the following instructions to setup my ES instance: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode I am able to play around with my instance on Kibana at http://0.0.0.0:5601/app/dev_tools#/console. Everything works upto here. Now I am trying to define some sample documents using django models and index them through the library; I am following the instructions here: https://django-elasticsearch-dsl.readthedocs.io/en/latest/quickstart.html#install-and-configure First, I add pip installed and added django_elasticsearch_dsl to INSTALLED_APPS Next, I added in settings.py: ELASTICSEARCH_DSL = { 'default': { 'hosts': 'localhost:9200' }, }  Then I create a sample model and document that looks like this: # models.py from django.db import models class Car(models.Model): name = models.CharField(max_length=30) color = models.CharField(max_length=30) description = models.TextField() type = models.IntegerField(choices=[ (1, "Sedan"), (2, "Truck"), (4, "SUV"), ])  # documents.py from django_elasticsearch_dsl import Document from django_elasticsearch_dsl.registries import registry from .models import Car @registry.register_document class CarDocument(Document): class Index: # Name of the Elasticsearch index name = 'cars' # See Elasticsearch Indices API reference for available settings settings = {'number_of_shards': 1, 'number_of_replicas': 0} class Django: model = Car # The model associated with this Document # The fields of the model you want to be indexed in Elasticsearch fields = [ 'name', 'color', 'description', 'type', ]  Finally, running python3 manage.py search_index --rebuild results in the following connection error:  raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))  I suspect that there might be an issue with my ELASTICSEARCH_DSL setup as I have not specified any config for https but the documentation does not make this clear. How do I resolve this issue? Django version: Django==4.0.1 django-elasticsearch-dsl==7.2.2 Python version: Python 3.9.10  Thanks!

Answer: I figured that it was an issue with my certificate. I needed to add some additional config param to the ELASTICSEARCH_DSL variable. Adding this solves the issue: from elasticsearch import RequestsHttpConnection # Elasticsearch configuration in settings.py ELASTICSEARCH_DSL = { 'default': { 'hosts': 'localhost:9200', 'use_ssl': True, 'http_auth': ('user', 'password'), 'ca_certs': '/path/to/cert.crt' 'connection_class': RequestsHttpConnection } }  See this section of the elastic docs on verifying a certificate. If you haven't setup a certificate to authenticate the connection with and just need to get something up and running quickly, you can pass 'verify_certs': False and set 'ca_certs' to None.
Question: Example of numbers  I am using the standard pytesseract img to text. I have tried with digits only option 90% of the time it is perfect but above is a example where it goes horribly wrong! This example produced no characters at all As you can see there are now letters so language option is of no use, I did try adding some text in the grabbed image but it still goes wrong. I increased the contrast using CV2 the text has been blurred upstream of my capture Any ideas on increasing accuracy? After many tests using the suggestions below. I found the sharpness filter gave unreliable results. another tool you can use is contrast=cv2.convertScaleAbs(img2,alpha=2.5,beta=-200) I used this as my text in black and white ended up light gray text on a gray background with convertScaleAbs I was able to increase the contrast to get almost a black and white image Basic steps for OCR  Convert to monochrome Crop image to your target text Filter image to get black and white perform OCR 

Answer: Here's a simple approach using OpenCV and Pytesseract OCR. To perform OCR on an image, it's important to preprocess the image. The idea is to obtain a processed image where the text to extract is in black with the background in white. To do this, we can convert to grayscale, then apply a sharpening kernel using cv2.filter2D() to enhance the blurred sections. A general sharpening kernel looks like this: [[-1,-1,-1], [-1,9,-1], [-1,-1,-1]]  Other kernel variations can be found here. Depending on the image, you can adjust the strength of the filter. From here we Otsu's threshold to obtain a binary image then perform text extraction using the --psm 6 configuration option to assume a single uniform block of text. Take a look here for more OCR configuration options.  Here's a visualization of the image processing pipeline: Input image  Convert to grayscale -> apply sharpening filter  Otsu's threshold  Result from Pytesseract OCR 124,685  Code import cv2 import numpy as np import pytesseract pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe" # Load image, grayscale, apply sharpening filter, Otsu's threshold image = cv2.imread('1.png') gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) sharpen_kernel = np.array([[-1,-1,-1], [-1,9,-1], [-1,-1,-1]]) sharpen = cv2.filter2D(gray, -1, sharpen_kernel) thresh = cv2.threshold(sharpen, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] # OCR data = pytesseract.image_to_string(thresh, lang='eng', config='--psm 6') print(data) cv2.imshow('sharpen', sharpen) cv2.imshow('thresh', thresh) cv2.waitKey()
Question: Tqdm documentation shows an example of tqdm working on pandas apply using progress_apply. I adapted the following code from here https://tqdm.github.io/docs/tqdm/ on a process that regularly take several minutes to perform (func1 is a regex function). from tqdm import tqdm tqdm.pandas() df.progress_apply(lambda x: func1(x.textbody), axis=1)  The resulting progress bar doesn't show any progress. It just jumps from 0 at the start of the loop to 100 when it is finished. I am currently running tqdm version 4.61.2

Answer: Utilizing tqdm with pandas Generally speaking, people tend to use lambdas when performing operations on a column or row. This can be done in a number of ways.  Please note: that if you are working in jupyter notebook you should use tqdm_notebook instead of tqdm. Also I'm not sure what your code looks like but if you're simply following the example given in the tqdm docs, and you're only performing 100 interations, computers are fast and will blow through that before your progress bar has time to update. Perhaps it would be more instructive to use a larger dataset like I provided below.  Example 1: from tqdm import tqdm # version 4.62.2 import pandas as pd # version 1.4.1 import numpy as np tqdm.pandas(desc='My bar!') # lots of cool paramiters you can pass here. # the below line generates a very large dataset for us to work with. df = pd.DataFrame(np.random.randn(100000000, 4), columns=['a','b','c','d']) # the below line will square the contents of each element in an column-wise # fashion df.progress_apply(lambda x: x**2)  Output: Output Example 2: # you could apply a function within the lambda expression for more complex # operations. And keeping with the above example... tqdm.pandas(desc='My bar!') # lots of cool paramiters you can pass here. # the below line generates a very large dataset for us to work with. df = pd.DataFrame(np.random.randn(100000000, 4), columns=['a','b','c','d']) def function(x): return x**2 df.progress_apply(lambda x: function(x))
Question: I have been trying to make a python script to zip a file with the zipfile module. Although the text file is made into a zip file, It doesn't seem to be compressing it; testtext.txt is 1024KB whilst testtext.zip (The code's creation) is also equal to 1024KB. However, if I compress testtext.txt manually in File Explorer, the resulting zip file is compressed (To 2KB, specifically). How, if possible, can I combat this logical error? Below is the script that I have used to (unsuccessfully) zip a text file. from zipfile import ZipFile textFile = ZipFile("compressedtextstuff.zip", "w") textFile.write("testtext.txt") textFile.close() 

Answer: Well that's odd. Python's zipfile defaults to the stored compression method, which does not compress! (Why would they do that?) You need to specify a compression method. Use ZIP_DEFLATED, which is the most widely supported. import zipfile zip = zipfile.ZipFile("stuff.zip", "w", zipfile.ZIP_DEFLATED) zip.write("test.txt") zip.close()
Question: I am trying to add type annotations/hints in a Python script for running mypy checks. I have a pandas.DataFrame object, which I iterate like this: someTable: pandas.DataFrame = pandas.DataFrame() # ... # adding some data to someTable # ... for index, row in someTable.iterrows(): #reveal_type(index) print(type(index)) print(index + 1)  If I run this script, here's what I get: $ python ./some.py <class 'int'> 2 <class 'int'> 3  And if I check it with mypy, then it reports errors: $ mypy ./some.py some.py:32: note: Revealed type is "Union[typing.Hashable, None]" some.py:34: error: Unsupported operand types for + ("Hashable" and "int") some.py:34: error: Unsupported operand types for + ("None" and "int") some.py:34: note: Left operand is of type "Optional[Hashable]" Found 2 errors in 1 file (checked 1 source file)  As I understand, mypy sees the index as Union[typing.Hashable, None], which is not int, and so index + 1 looks like an error to it. How and where should I then annotate/hint it to satisfy mypy? I tried this: index: int for index, row in someTable.iterrows(): # ...  but that results in: $ mypy ./some.py some.py:32: error: Incompatible types in assignment (expression has type "Optional[Hashable]", variable has type "int") Found 1 error in 1 file (checked 1 source file) 

Answer: You could hint index as Optional[int], but then x + 1 won't type check. I'm not sure where Union[typing.Hashable, None] comes from; iterrows itself returns an Iterable[tuple[Hashable, Series]]. But it seems like you can safely assert that if index is assigned a value, then it will not be None. index: Optional[int] for index, row in someTable.iterrows(): index = typing.cast(int, index) print(index + 1)  (Is the Union supposed to reflect the possibility of the iterable raising StopIteration? That doesn't seem right, as a function that raises an exception doesn't return None; it doesn't return at all.)
Question: I am trying to make an application using python and gRPC as shown in this article - link I am able to run the app successfully on my terminal but to run with a frontend I need to run it as a flask app, codebase. And I am doing all this in a virtual environment. when I run my flask command FLASK_APP=marketplace.py flask run This is the error I get ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/Users/alex/Desktop/coding/virt/lib/python3.8/site-packages/markupsafe/__init__.py)  On researching about this error I found this link - it basically tells us that currently I am using a higher version of MarkUpSafe library than required. So I did pip freeze --local  inside the virtualenv and got MarkUpSafe version to be MarkupSafe==2.1.0 I think if I change the version of this library from 2.1.0 to 2.0.1 then the flask app might run. How can I change this library's version from the terminal? PS: If you think changing the version of the library won't help in running the flask app, please let me know what else can I try in this.

Answer: If downgrading will solve the issue for you try the following code inside your virtual environment. pip install MarkupSafe==2.0.1