eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a three-class problem and I'm able to report precision and recall for each class with the below code: from sklearn.metrics import classification_report print(classification_report(y_test, y_pred))  which gives me the precision and recall nicely for each of the 3 classes in a table format. My question is how can I now get sensitivity and specificity for each of the 3 classes? I looked at sklearn.metrics and I didn't find anything for reporting sensitivity and specificity.

Answer: If we check the help page for classification report:  Note that in binary classification, recall of the positive class is also known as “sensitivity”; recall of the negative class is “specificity”.  So we can convert the pred into a binary for every class, and then use the recall results from precision_recall_fscore_support. Using an example: from sklearn.metrics import classification_report y_true = [0, 1, 2, 2, 2] y_pred = [0, 0, 2, 2, 1] target_names = ['class 0', 'class 1', 'class 2'] print(classification_report(y_true, y_pred, target_names=target_names))  Looks like:  precision recall f1-score support class 0 0.50 1.00 0.67 1 class 1 0.00 0.00 0.00 1 class 2 1.00 0.67 0.80 3 accuracy 0.60 5 macro avg 0.50 0.56 0.49 5 weighted avg 0.70 0.60 0.61 5  Using sklearn: from sklearn.metrics import precision_recall_fscore_support res = [] for l in [0,1,2]: prec,recall,_,_ = precision_recall_fscore_support(np.array(y_true)==l, np.array(y_pred)==l, pos_label=True,average=None) res.append([l,recall[0],recall[1]])  put the results into a dataframe: pd.DataFrame(res,columns = ['class','sensitivity','specificity']) class sensitivity specificity 0 0 0.75 1.000000 1 1 0.75 0.000000 2 2 1.00 0.666667
Question: i know how to iterate over items in dictionary using for loop. but i need know is how to iterate over items in dictionary using while loop. is that possible? This how i tried it with for loop. user_info = { "username" : "Hansana123", "password" : "1234", "user_id" : 3456, "reg_date" : "Nov 19" } for values,keys in user_info.items(): print(values, "=", keys) 

Answer: You can iterate the items of a dictionary using iter and next with a while loop. This is almost the same process as how a for loop would perform the iteration in the background on any iterable.  https://docs.python.org/3/library/functions.html https://docs.python.org/3/library/stdtypes.html#iterator-types  Code: user_info = { "username" : "Hansana123", "password" : "1234", "user_id" : 3456, "reg_date" : "Nov 19" } print("Using for loop...") for key, value in user_info.items(): print(key, "=", value) print() print("Using while loop...") it_dict = iter(user_info.items()) while key_value := next(it_dict, None): print(key_value[0], "=", key_value[1])  Output: Using for loop... username = Hansana123 password = 1234 user_id = 3456 reg_date = Nov 19 Using while loop... username = Hansana123 password = 1234 user_id = 3456 reg_date = Nov 19
Question: I am using the caplog fixture that comes with pytest. I am using mypy for my type checking, and would like to know what the correct type hint for caplog is. For example: def test_validate_regs(caplog: Any) -> None: validate_regs(df, logger) assert caplog.text == "", "No logs should have been made."  In this example, I have it set to Any, but I am wondering if there is a more specific type hint I can use. I've tried reading the docs on caplog, as well as searching the pytest code in github to see what the caplog fixture returns, but couldn't find anything more than the following. But using a str type just gave me an error, saying that str type doesn't have the attribute text, which makes sense. When I print the type of caplog I get _pytest.logging.LogCaptureFixture although I'm not sure how I'd import this from _pytest and make use of it.

Answer: as of pytest 6.2.0 you should use pytest.LogCaptureFixture prior to that you needed to import a private name which is not recommended (we frequently change the internals inside the _pytest namespace without notice and with no promises of forward or backward compatibility)  disclaimer: I'm a pytest core dev
Question: I am trying to build a control structure in a class method that takes a function as input and has different behaviors if a function is decorated or not. Any ideas on how you would go about building a function is_decorated that behaves as follows: def dec(fun): # do decoration def func(data): # do stuff @dec def func2(data): # do other stuff def is_decorated(func): # return True if func has decorator, otherwise False is_decorated(func) # False is_decorated(func2) # True 

Answer: Yes, it's relatively easy because functions can have arbitrary attributes added to them, so the decorator function can add one when it does its thing: def dec(fun): def wrapped(*args, **kwargs): pass wrapped.i_am_wrapped = True return wrapped def func(data): ... # do stuff @dec def func2(data): ... # do other stuff def is_decorated(func): return getattr(func, 'i_am_wrapped', False) print(is_decorated(func)) # -> False print(is_decorated(func2)) # -> True
Question: Hi so i have code which gives the answer in form of speech. I am using this code: audio.save("audio.wav") sound_file = '/content/audio.wav' Audio(sound_file, autoplay=True)  Now the code plays file all well but if I play in a separate cell. But if I put this code in between my code, it doesn't work. Any ideas?

Answer: Wrap it in display(): from IPython.display import Audio, display display(Audio(sound_file, autoplay=True))
Question: I have two dataframes: data = { 'values': ['Cricket', 'Soccer', 'Football', 'Tennis', 'Badminton', 'Chess'], 'gems': ['A1K, A2M, JA3, AN4', 'B1, A1, Bn2, B3', 'CD1, A1', 'KWS, KQM', 'JP, CVK', 'KF, GF'] } df1 = pd.DataFrame(data)  df1  values gems 0 Cricket A1K, A2M, JA3, AN4 1 Soccer B1, A1, Bn2, B3 2 Football CD1, A1 3 Tennis KWS, KQM 4 Badminton JP, CVK 5 Chess KF, GF  second dataframe data2 = { '1C': ['B1', 'K1', 'A1K', 'J1', 'A4'], '02C': ['Bn2', 'B3', 'JK', 'ZZ', 'ko'], '34C': ['KF', 'CD1', 'B3','ji', 'HU'] } df2 = pd.DataFrame(data2)  df2  1C 02C 34C 0 B1 Bn2 KF 1 K1 B3 CD1 2 A1K JK B3 3 J1 ZZ ji 4 A4 ko HU  I want check items in df1['gems'] in each column of df2 and represent their counts and overlapping items. The expected output is:  values gems 1C 1CGroup 02C 02CGroup 34C 34CGroup 0 Cricket A1K, A2M, JA3, AN4 1 A1K 0 NA 0 NA 1 Soccer B1, A1, Bn2, B3 1 Bn2 2 Bn2, B3 1 B3 2 Football CD1, A1 0 NA 0 NA 1 CD1 3 Tennis KWS, KQM 0 NA 0 NA 0 NA 4 Badminton JP, CVK 0 NA 0 NA 0 NA 5 Chess KF, GF 0 NA 0 NA 1 KF 

Answer: first str.split and explode the column gems and reset_index to keep the original index. Then for each column of df2, merge with the exploded gems, groupby the original index and do both the count and the aggregation as you want with join. pd.concat the merges for each column and join to your original df1. fillna the count columns with 0 as shown in the expected output. # one row per gem used in the merge df_ = df1['gems'].str.split(', ').explode().reset_index() res = ( df1.join( #can join to df1 as we keep the original index value pd.concat([df_.merge(df2[[col]], left_on='gems', right_on=col) .groupby('index') # original index in df1 [col].agg(**{col: 'count', # do each aggregation f'{col}Group':lambda x: ', '.join(x)}) for col in df2.columns], # do it for each column of df2 axis=1)) .fillna({col:0 for col in df2.columns}) #fill the count columns with 0 ) print(res) values gems 1C 1CGroup 02C 02CGroup 34C 34CGroup 0 Cricket A1K, A2M, JA3, AN4 1.0 A1K 0.0 NaN 0.0 NaN 1 Soccer B1, A1, Bn2, B3 1.0 B1 2.0 Bn2, B3 1.0 B3 2 Football CD1, A1 0.0 NaN 0.0 NaN 1.0 CD1 3 Tennis KWS, KQM 0.0 NaN 0.0 NaN 0.0 NaN 4 Badminton JP, CVK 0.0 NaN 0.0 NaN 0.0 NaN 5 Chess KF, GF 0.0 NaN 0.0 NaN 1.0 KF
Question: On a fresh ubunto i installed docker and when i run the image, i got following error docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "--gpus": executable file not found in $PATH: unknown. ERRO[0000] error waiting for container: context canceled  This is how i make system ready sudo apt updat sudo apt full-upgrade sudo apt autoremove  #install gpu drivers via software and updates sudo apt install nvidia-cuda-toolkit sudo apt install docker.io  #here is docker file FROM python:3 WORKDIR /workspace COPY test.py /workspace RUN pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html CMD ["python", "./test.py"]  #here is test.py file import torch print('testing') print(torch.cuda.get_device_name(0)) print(torch.cuda.is_available()) print(torch.cuda.current_device()) print(torch.cuda.device_count()) print('gpu')  nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:07:16_PDT_2019 Cuda compilation tools, release 10.1, V10.1.243  nvidia-smi -----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A | | N/A 64C P0 24W / N/A | 513MiB / 6069MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+  uname -r 5.8.0-63-generic lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal  docker version Client: lient: Version: 20.10.2 API version: 1.41 Go version: go1.13.8 Git commit: 20.10.2-0ubuntu1~20.04.3 Built: Fri Jul 23 21:06:26 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.2 API version: 1.41 (minimum version 1.12) Go version: go1.13.8 Git commit: 20.10.2-0ubuntu1~20.04.3 Built: Fri Jul 23 19:35:35 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.2-0ubuntu1~20.04.2 GitCommit: runc: Version: 1.0.0~rc95-0ubuntu1~20.04.2 GitCommit: docker-init: Version: 0.19.0 GitCommit:  This is how i build image and run it sudo docker build -t test . sudo docker run test --gpus all 

Answer: This command is incorrectly ordered: sudo docker run test --gpus all  The docker run command takes the syntax: docker ${args_to_docker} run ${args_to_run} image_name ${cmd_override}  The --gpus is a flag to the run command, and not a command you want to run inside your container. So you'd reorder as: sudo docker run --gpus all test
Question: I'm using FactoryBoy and Faker to generate some models for unit tests. Generating data for fields is easy enough, but how to I generate a string that incorporates a value produced from a Faker provider? import factory import MyModel class MyFactory(factory.django.DjangoModelFactory): class Meta: model = MyModel # my_ip will be a temporary variable that is not returned by the factory. exclude = (my_ip,) my_ip = factory.Faker("ipv4_private") my_string = f"String with IP address [{my_ip}]"  Using MyFactory then creates objects with my_string set to something like: String with IP address [<factory.faker.Faker object at 0x7fc656aa6358>] How can I get my_string to contain something like: String with the IP address [192.168.23.112] How do I resolve this to a value, rather than just getting the object? I've tried wrapping in str() with no luck. Do I need to use LazyAttribute or LazyFunction or something from FactoryBoy?

Answer: With some trial and error I figured out this did require using LazyAttribute so the my_string attribute is calculated after the rest of the object is generated. However, I then discovered this is essentially a duplicate of: In Factory Boy, how to join strings created with Faker? If anyone is wondering, the way to do this is: my_string = factory.LazyAttribute(lambda o: f"String with IP address [{o.my_ip}]")
Question: I'm trying to use Tesseract-OCR to get the readings on below images but having issues getting consistent results with the spotted background. I have below configuration on my pytesseract CONFIG = f"—psm 6 -c tessedit_char_whitelist=01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄabcdefghijklmnopqrstuvwxyzåäö.,-" I have also tried below image pre-processing with some good results, but still not perfect results blur = cv2.blur(img,(4,4)) (T, threshInv) = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)  What I want is to consistently be able to identify the numbers and the decimal separator. What image pre-processing could help in getting consistent results on images as below?       

Answer: That was a challenge but i think i have an interesting approach: Pattern-matching If you zoom in, you realize that the pattern in the back only has 4 possible dots, a single full pixle, a double full pixel and a double pixel with a medium left or right. So what i did was grab these 4 patterns from the image with 17.160.000,00 and got to work. Save these to load again, i just grabbed them on the fly img = cv2.imread('C:/Users/***/17.jpg', cv2.IMREAD_GRAYSCALE) pattern_1 = img[2:5,1:5] pattern_2 = img[6:9,5:9] pattern_3 = img[6:9,11:15] pattern_4 = img[9:12,22:26] # just to show it carries over to other pics ;) img = cv2.imread('C:/Users/****/6.jpg', cv2.IMREAD_GRAYSCALE)  Actual Pattern Matching Next we match all the patterns and threshold to find all occurrences, i used 0.7 but you can play around with it a little. These patterns take off some pixels on the side and only match a sigle pixel on the left so we pad twice (one with an extra) to hit both for the first 3 patterns. The last one is the single pixel so it doesnt need it res_1 = cv2.matchTemplate(img,pattern_1,cv2.TM_CCOEFF_NORMED ) thresh_1 = cv2.threshold(res_1,0.7,1,cv2.THRESH_BINARY)[1].astype(np.uint8) pat_thresh_1 = np.pad(thresh_1,((1,1),(1,2)),'constant') pat_thresh_15 = np.pad(thresh_1,((1,1),(2,1)), 'constant') res_2 = cv2.matchTemplate(img,pattern_2,cv2.TM_CCOEFF_NORMED ) thresh_2 = cv2.threshold(res_2,0.7,1,cv2.THRESH_BINARY)[1].astype(np.uint8) pat_thresh_2 = np.pad(thresh_2,((1,1),(1,2)),'constant') pat_thresh_25 = np.pad(thresh_2,((1,1),(2,1)), 'constant') res_3 = cv2.matchTemplate(img,pattern_3,cv2.TM_CCOEFF_NORMED ) thresh_3 = cv2.threshold(res_3,0.7,1,cv2.THRESH_BINARY)[1].astype(np.uint8) pat_thresh_3 = np.pad(thresh_3,((1,1),(1,2)),'constant') pat_thresh_35 = np.pad(thresh_3,((1,1),(2,1)), 'constant') res_4 = cv2.matchTemplate(img,pattern_4,cv2.TM_CCOEFF_NORMED ) thresh_4 = cv2.threshold(res_4,0.7,1,cv2.THRESH_BINARY)[1].astype(np.uint8) pat_thresh_4 = np.pad(thresh_4,((1,1),(1,2)),'constant')  Editing the Image Now the only thing left to do is remove all the matches from the image. Since we have a mostly white backround we just set them to 255 to blend in. img[pat_thresh_1==1] = 255 img[pat_thresh_15==1] = 255 img[pat_thresh_2==1] = 255 img[pat_thresh_25==1] = 255 img[pat_thresh_3==1] = 255 img[pat_thresh_35==1] = 255 img[pat_thresh_4==1] = 255  Output  Edit: Take a look at Abstracts answer as well for refining this output and tesseract finetuning
Question: I have seen two ways for defining Abstract Classes in Python. This one: from abc import ABCMeta, abstractmethod class AbstactClass(metaclass = ABCMeta):  And this one: from abc import ABC, abstractmethod class AbstractClass2(ABC):  What are there differences between them and what are the respective usage scenarios?

Answer: There is no actual functional difference. The ABC class is simply a convenience class to help make the code look less confusing to those who don't know the idea of a metaclass well, as the documentation states:  A helper class that has ABCMeta as its metaclass. With this class, an abstract base class can be created by simply deriving from ABC avoiding sometimes confusing metaclass usage  which is even clearer if you look at the the implementation of abc.py, which is nothing more than an empty class that specifies ABCMeta as its metaclass, just so that its descendants can inherit the type: class ABC(metaclass=ABCMeta): """Helper class that provides a standard way to create an ABC using inheritance. """ __slots__ = ()
Question: I have alway used 'utf8' to read in a file:  with open(filename, 'r', encoding='utf8') as f, open(filename2, 'r', encoding='utf8') as f2: for line in f: line = line.strip() columns = line.split(' ') for line in f2: line = line.strip() columns = line.split(' ')  However, the code above introduced an additional '\ufeff' code at the line of for 'f2': columns = line.split(' ')  Now the columns[0] contains this character, while 'line' doesn't have this character. Why is that? Then I switched to 'utf-8-sig', and the problem is gone. However, the first file reading 'f' and the 'columns' doesn't have this issue at all even with 'encoding=utf8' only. Both are plain text files. So I have two questions regarding:  I am using Python3 and when reading a file, should I always use 'utf-8-sig' to be safe? Why doesn't 'line' contain this additional code, but 'columns' contains it? 

Answer: UTF-8-encoded files can be written with a signature indicating it is UTF-8. This signature code is called the "byte order mark" (or BOM) and has the Unicode code point value U+FEFF. If the file containing a BOM is viewed in a hex editor the file will start with the hexadecimal bytes EF BB BF. When viewed in a text editor with a non UTF-8 encoding they often appear as ï»¿ but that depends on the encoding. The 'utf-8-sig' codec can read UTF-8-encoded files written with and without the starting BOM signature and will remove it if present. Use 'utf-8-sig' for writing a file only if you want a UTF-8 BOM written at the start of the file. Some (usually Windows) programs, such as Excel when reading text files, expect a BOM if the file contains UTF-8, and assume a localized encoding otherwise. Other programs may not expect a BOM and could read it as an extra character, so the choice is yours. So for your two questions:  I am using Python3 and when reading a file, should I always use 'utf-8-sig' to be safe?  Yes, it will remove the BOM if present.  Why doesn't 'line' contain this additional code, but 'columns' contains it?  line.strip() doesn't remove \ufeff so I can't reproduce your claim. If a UTF-8 w/ BOM-encoded file is opened with utf8 the first character should be \ufeff. Are you using print to display the line? \ufeff is a whitespace character if printed: >>> line = '\ufeffabc' >>> line '\ufeffabc' >>> print(line) abc >>> print(line.strip()) abc >>> line.strip() '\ufeffabc'
Question: I'd like to be able to set timeout on subprocess stdout and return an empty string if exceeded timeout. Here's my attempt to do so using asyncio. However, it failed on using file.stdout.readline() in asyncio.wait_for. Any idea how to fix this ? import threading import select import subprocess import queue import time import asyncio class collector(): @staticmethod async def readline(file, timeout=3): try: line = await asyncio.wait_for(file.stdout.readline(), timeout) except asyncio.TimeoutError: return "" else: return line @staticmethod async def background(command): f = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) while True: line = await collector.readline(file=f, timeout=3) asyncio.run(collector.background("tail -f /tmp/2222"))  And here's callstack :  File "/tmp/script.py", line 13, in readline line = await asyncio.wait_for(file.stdout.readline(), timeout) File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 462, in wait_for fut = ensure_future(fut, loop=loop) File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 679, in ensure_future raise TypeError('An asyncio.Future, a coroutine or an awaitable is ' TypeError: An asyncio.Future, a coroutine or an awaitable is required 

Answer: The subprocess library only provides synchronous functions. These cannot be directly used by asyncio, and manually wrapping them is inefficient. asyncio already ships with its own subprocess backend. Its process representation works similar to subprocess.Popen but allows to cooperatively wait for operations. import asyncio.subprocess async def background(*command): # create subprocess via asyncio proc = await asyncio.create_subprocess_exec( *command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) while True: line = await readline(proc.stdout, timeout=0.01) print("read", len(line), "characters") # read from an async stream vvvvvvvvvvvvvvvvvv instead of a file-like object async def readline(stream: asyncio.StreamReader, timeout: float): try: # stream.readline is a coroutine vvvvvvvvvvvv return await asyncio.wait_for(stream.readline(), timeout=timeout) except asyncio.TimeoutError: return "" asyncio.run(background("cat", "/dev/random"))
Question: What I am trying to do I am using PyArrow to read some CSVs and convert them to Parquet. Some of the files I read have plenty of columns and have a high memory footprint (enough to crash the machine running the job). I am trying to chunk through the file while reading the CSV in a similar way to how Pandas read_csv with chunksize works. For example this is how the chunking code would work in pandas: chunks = pandas.read_csv(data, chunksize=100, iterator=True) # Iterate through chunks for chunk in chunks: do_stuff(chunk)  I want to port a similar functionality to Arrow What I have tried to do I noticed that Arrow has ReadOptions which include a block_size parameter, and I thought maybe I could use it like: # Reading in-memory csv file arrow_table = arrow_csv.read_csv( input_file=input_buffer, read_options=arrow_csv.ReadOptions( use_threads=True, block_size=4096 ) ) # Iterate through batches for batch in arrow_table.to_batches(): do_stuff(batch)  As this (block_size) does not seem to return an iterator, I am under the impression that this will still make Arrow read the entire table in memory and thus recreate my problem. Lastly, I am aware that I can first read the csv using Pandas and chunk through it then convert to Arrow tables. But I am trying to avoid using Pandas and only use Arrow. I am happy to provide additional information if needed

Answer: The function you are looking for is pyarrow.csv.open_csv which returns a pyarrow.csv.CSVStreamingReader. The size of the batches will be controlled by the block_size option you noticed. For a complete example: import pyarrow as pa import pyarrow.parquet as pq import pyarrow.csv in_path = '/home/pace/dev/benchmarks-proj/benchmarks/data/nyctaxi_2010-01.csv.gz' out_path = '/home/pace/dev/benchmarks-proj/benchmarks/data/temp/iterative.parquet' convert_options = pyarrow.csv.ConvertOptions() convert_options.column_types = { 'rate_code': pa.utf8(), 'store_and_fwd_flag': pa.utf8() } writer = None with pyarrow.csv.open_csv(in_path, convert_options=convert_options) as reader: for next_chunk in reader: if next_chunk is None: break if writer is None: writer = pq.ParquetWriter(out_path, next_chunk.schema) next_table = pa.Table.from_batches([next_chunk]) writer.write_table(next_table) writer.close()  This example also highlights one of the challenges the streaming CSV reader introduces. It needs to return batches with consistent data types. However, when parsing CSV you typically need to infer the data type. In my example data the first few MB of the file have integral values for the rate_code column. Somewhere in the middle of the batch there is a non-integer value (* in this case) for that column. To work around this issue you can specify the types for columns up front as I am doing here.