eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am doing performance/memory analysis on a certain method that is wrapped with the functools.lru_cache decorator. I want to see how to inspect the current size of my cache without doing some crazy inspect magic to get to the underlying cache. Does anyone know how to see the current cache size of method decorated with functools.lru_cache?

Answer: Digging around in the docs showed the answer is calling .cache_info() on the method.  To help measure the effectiveness of the cache and tune the maxsize parameter, the wrapped function is instrumented with a cache_info() function that returns a named tuple showing hits, misses, maxsize and currsize. In a multi-threaded environment, the hits and misses are approximate.
Question: I'm creating my first test script (yay!) I'm have a list of dictionaries where one of the keys is a list. I'd like the test to past if the list (in the dictionary) is in any order. I know you can use assertCountEqual to check list equality regardless of order, but can you do that for a list that contains a dictinary of lists? See below for example Will Succeed def test(self): output = [2,1] desired_output = [1,2] self.assertCountEqual(output, desired_output)  Will Fail def test(self): desired_output = [{'count': 2, 'columns': ['col2', 'col5']}] output = [{'count': 2, 'columns': ['col5', 'col2']}] self.assertCountEqual(output, desired_output)  Thanks

Answer: The assertCountEqual(first, second, msg=None):  Test that sequence first contains the same elements as second, regardless of their order. When they don’t, an error message listing the differences between the sequences will be generated.  Important: Calling assertCountEqual(first, second, msg=None) is equivalent to callingassertEqual(Counter(list(first)), Counter(list(second))). Note: A Counter is a dict subclass for counting hashable objects. It is a collection where elements are stored as dictionary keys and their counts are stored as dictionary values. For this to work the keys have to be hashable but unfortunately the dict is unhashable because its mutable.  In order to perform the desired task you could use the frozenset. The frozenset builds an immutable unordered collection of unique elements. In order to have the test succeed you will have to build a dict whose values corresponding to its keys are immutable. We can use the recursive approach to build a dictionary that contains immutable values. Try this (UPDATE): def getHashableDict(dictionary): hashable_dict = {} for key, value in dictionary.items(): if isinstance(value, list): hashable_dict[key] = frozenset(value) elif isinstance(value, dict): hashable_dict[key] = getHashableDict(value) else: hashable_dict[key] = value return frozenset(hashable_dict.items()) def test(self): desired_output = [{'count': 2, 'columns': ['col2', 'col5']}] output = [{'count': 2, 'columns': ['col5', 'col2']}] output = [getHashableDict(item) for item in output] #--> create list of hashable types desired_output = [getHashableDict(item) for item in desired_output] self.assertCountEqual(output, desired_output)  The test will now succeed.
Question: We are using a proxy + profile when using the aws s3 commands to browse our buckets in CLI. export HTTPS_PROXY=https://ourproxyhost.com:3128 aws s3 ls s3://our_bucket/.../ --profile dev  And we can work with our buckets and objects fine. Because I need to write Python code for this, I translated this using boto3: # python 2.7.12 import boto3 # v1.5.18 from botocore.config import Config # v1.8.32 s3 = boto3.Session(profile_name='dev').resource('s3', config=Config(proxies={'https': 'ourproxyhost.com:3128'})).meta.client obj = s3.get_object(Bucket='our_bucket', Key='dir1/dir2/.../file')  What I get is this: botocore.vendored.requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)  Why is this working in CLI, but not in Python?

Answer:  botocore.vendored.requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)  The error above in most cases it's usually related to the CA bundle being used for S3 connections. Possible Resolution Steps: 1. Turn off SSL certification validation : s3 = boto3.client('s3', verify=False)  As mentioned in this boto3 documentation, this option turns off validation of SSL certificates but SSL protocol will still be used (unless use_ssl is False) for communication. 2. Check if you have AWS_CA_BUNDLE env var set?: echo $AWS_CA_BUNDLE  or  export | grep AWS_CA_BUNDLE  3. Check if you have certifi installed in your python env?: pip list | grep certifi  Depending on the output of the above command, you could be using a version of certifi (which is not a dependency of boto3) that has a broken certificate validation when communicating with s3 endpoints. You will need to upgrade your OpenSSL version or pin certifi to a stable version as shown below : sudo pip uninstall certifi sudo pip install certifi==2015.04.28  Hope this helps!
Question: Dataframe X: A B C D V1 V2 V3 V4 V1 V3 V4 V5 V1 V4 V5 V5 V1 V5 V9 V5 V1 V2 V3 V4 V1 V10 V11 V12 V1 V10 V6 V8 V1 V12 V7 V8  Here Col A has 1 unique value, Col B has 6 unique values, Col C has 7 unique values, Col D has 4 unique values. I need a list of all columns where unique values > 4 say. X.columns[(X.nunique() > 4).any()]  I expect to get only col B and Col C here, but I get all columns. How to achieve desired output.

Answer: You are really close, only remove .any for boolean mask: c = X.columns[(X.nunique() > 4)] print (c) Index(['B', 'C'], dtype='object')  If need select columns use DataFrame.loc: df = X.loc[:, (X.nunique() > 4)] print (df) B C 0 V2 V3 1 V3 V4 2 V4 V5 3 V5 V9 4 V2 V3 5 V10 V11 6 V10 V6 7 V12 V7
Question: I am working on a python project that does some analysis on csv files. I know there is no well-definedstandard for csv files, but as far as I understood the definition (https://www.rfc-editor.org/rfc/rfc4180#page-2), I think that a csv file should not contain more than one table. Is this thinking correct, or did I misunderstood the definitions? How often do you see more than one table in csv's?

Answer: You are correct. There is no universal accepted standard. The definition is written to suggest that each file contains one table, and this is by far the most common practice. There's technically nothing stopping you from having more than one table, using a format you decide on and implement and keep consistent. For instance, you could parse the file yourself and use a line with 5 hyphens to designate a separate table. However I wouldn't recommend this. It goes against the common practice, and you will eliminate the possibility of using existing CSV libraries to help you.
Question: I wrote a class that inherits another class: class ValueSum(SubQuery): output = IntegerField()  And pycharm is showing the following warning:  Class ValueSum must implement all abstract methods  Then I alt+enter to add ABC to superclass. And my warning is gone. I have several questions:  Should I always do this when writing a sub-class?  What is the difference between manually implementing all the methods vs just using ABC? Does ABC add something to my code? 

Answer: SubQuery is an abstract base class (per the abc module) with one or more abstract methods that you did not override. By adding ABC to the list of base classes, you defined ValueSum itself to be an abstract base class. That means you aren't forced to override the methods, but it also means you cannot instantiate ValueSum itself. PyCharm is warning you ahead of time that you need to implement the abstract methods inherited from SubQuery; if you don't, you would get an error from Python when you actually tried to instantiate ValueSum.  As to what inheriting from ABC does, the answer is... not much. It's a convenience for setting the metaclass. The following are equivalent: class Foo(metaclass=abc.ABCMeta): ...  and class Foo(abc.ABC): ...  The metaclass modifies __new__ so that every attempt to create an instance of your class checks that the class has implemented all methods decorated with @abstractmethod in a parent class.
Question: Enums can be created by taking in a list of possible members, which I'm doing like so: # example_issue.py import enum yummy_foods = ["ham", "cheese"] foods = enum.Enum("Foods", yummy_foods) cheese = foods.cheese  This looks OK, and runs fine, but mypy returns example_issue.py:4: error: Enum() expects a string, tuple, list or dict literal as the second argument example_issue.py:5: error: "Type[foods]" has no attribute "cheese" Found 2 errors in 1 file (checked 1 source file)  What is mypy doing here, and why can't it follow that foods can take any value in yummy_foods?

Answer: Using a variable yummy_foods is too dynamic for mypy's static type checking, see this GitHub issue. If you change your code to generate the Enum as: foods = enum.Enum("Foods", ["ham", "cheese"])  mypy will then be able to figure out which attributes exist on the enumeration.
Question: I ran a command in python pandas as follows: q1_fisher_r[(q1_fisher_r['TP53']==1) & q1_fisher_r[(q1_fisher_r['TumorST'].str.contains(':1:'))]]  I got following error: TypeError: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool]  the solution i tried using this: error link. changed the code accordingly as: q1_fisher_r[(q1_fisher_r['TumorST'].str.contains(':1:')) & (q1_fisher_r[(q1_fisher_r['TP53']==1)])]  But still I got the same error as TypeError: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool]

Answer: For filtering by multiple conditions chain them by & and filter by boolean indexing: q1_fisher_r[(q1_fisher_r['TP53']==1) & q1_fisher_r['TumorST'].str.contains(':1:')] ^^^^ ^^^^ first condition second condition  Problem is this code returned filtered data, so cannot chain by condition: q1_fisher_r[(q1_fisher_r['TumorST'].str.contains(':1:'))]  Similar problem: q1_fisher_r[(q1_fisher_r['TP53']==1)]  Sample: q1_fisher_r = pd.DataFrame({'TP53':[1,1,2,1], 'TumorST':['5:1:','9:1:','5:1:','6:1']}) print (q1_fisher_r) TP53 TumorST 0 1 5:1: 1 1 9:1: 2 2 5:1: 3 1 6:1 df = q1_fisher_r[(q1_fisher_r['TP53']==1) & q1_fisher_r['TumorST'].str.contains(':1:')] print (df) TP53 TumorST 0 1 5:1: 1 1 9:1:
Question: Let's say we have a function definition like this: def f(*, model: Optional[Type[pydantic.BaseModel]] = None)  So the function doesn't require pydantic to be installed until you pass something as a model. Now let's say we want to pack the function into pypi package. And my question is if there's a way to avoid bringing pydantic into the package dependencies only the sake of type checking?

Answer: I tried to follow dspenser's advice, but I found mypy still giving me Name 'pydantic' is not defined error. Then I found this chapter in the docs and it seems to be working in my case too: from typing import TYPE_CHECKING if TYPE_CHECKING: import pydantic  You can use normal clases (instead of string literals) with __future__.annotations (python 3.8.1): from __future__ import annotations from typing import TYPE_CHECKING, Optional, Type if TYPE_CHECKING: import pydantic def f(*, model: Optional[Type[pydantic.BaseModel]] = None): pass  If for some reason you can't use __future__.annotations, e.g. you're on python < 3.7, use typing with string literals from dspenser's solution.
Question: I want to accumulate a sum using multiprocessing.Pool. Here's how I tried: import multiprocessing def add_to_value(addend, value): value.value += addend with multiprocessing.Manager() as manager: value = manager.Value(float, 0.0) with multiprocessing.Pool(2) as pool: pool.starmap(add_to_value, [(float(i), value) for i in range(100)]) print(value.value)  This gives incorrect and even inconsistent results. For instance, one time it gives 2982.0 and another it gives 2927.0. The correct output is 4950.0, and I do get this when I use only one process in the call to Pool, rather than 2. I'm using Python 3.7.5.

Answer: The multiprocessing documentation (under multiprocessing.Value) is quite explicit about this:  Operations like += which involve a read and write are not atomic. So if, for instance, you want to atomically increment a shared value it is insufficient to just do counter.value += 1.  In short, you need to grab a lock to be able to do this. You can do that with: def add_to_value(addend, value, lock): with lock: value.value += addend if __name__ == '__main__': with multiprocessing.Manager() as manager: lock = manager.Lock() value = manager.Value(float, 0.0) with multiprocessing.Pool(2) as pool: pool.starmap(add_to_value, [(float(i), value, lock) for i in range(100)]) print(value.value)  This will correctly output 4950.0. But note that this approach will be quite expensive due to the need for locking. Most probably, it will take more time to finish than if you have a single process doing the operation. NOTE: I'm also adding an if __name__ == '__main__': guard which is actually required when using a start method other than fork. The default on both Windows and Mac OS is spawn, so that's really needed to make this code portable to either of those platforms. Start methods spawn and forkserver are also available on Linux/Unix, so in some situations this is also needed there. Multiprocessing will be more efficient when you're able to offload a job to workers that they can complete on their own, for example calculate partial sums and then add them together in the main process. If possible, consider rethinking your approach to fit that model.
Question: I'm trying to map a dataset to a blank CSV file with different headers, so I'm essentially trying to map data from one CSV file which has different headers to a new CSV with different amount of headers and called different things, the reason this question is different is since the column names aren't the same but there are no overlapping columns either. And I can't overwrite the data file with new headers since the data file has other columns with irrelevant data, I'm certain I'm overcomplicating this. I've seen this example code but how do I change this since this example is using a common header to join the data. a = pd.read_csv("a.csv") b = pd.read_csv("b.csv") #a.csv = ID TITLE #b.csv = ID NAME b = b.dropna(axis=1) merged = a.merge(b, on='title') merged.to_csv("output.csv", index=False)  Sample Data a.csv (blank format file, the format must match this file): Headers: TOWN NAME LOCATION HEIGHT STAR  b.csv: Headers: COUNTRY WEIGHT NAME AGE MEASUREMENT Data: UK, 150lbs, John, 6, 6ft  Expected output file: Headers: TOWN NAME LOCATION HEIGHT STAR Data: (Blank) John, UK, 6ft (Blank) 

Answer: From your example, it looks like you need to do some column renaming in addition to the merge. This is easiest done before the merge itself.  # Read the csv files dfA = pd.read_csv("a.csv") dfB = pd.read_csv("b.csv") # Rename the columns of b.csv that should match the ones in a.csv dfB = dfB.rename(columns={'MEASUREMENT': 'HEIGHT', 'COUNTRY': 'LOCATION'}) # Merge on all common columns df = pd.merge(dfA, dfB, on=list(set(dfA.columns) & set(dfB.columns)), how='outer') # Only keep the columns that exists in a.csv df = df[dfA.columns] # Save to a new csv df.to_csv("output.csv", index=False)  This should give you what you are after.
Question: I have a problem with choosing right parameters for HoughCircles function. I try to detect circles from video. This circles are made by me, and has almost the same dimension. Problem is that camera is in move.  When I change maxRadius it still detect bigger circles somehow (see the right picture). I also tried to change param1, param2 but still no success.  gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) blurred = cv2.medianBlur(gray, 25)#cv2.bilateralFilter(gray,10,50,50) minDist = 100 param1 = 500 param2 = 200#smaller value-> more false circles minRadius = 5 maxRadius = 10 circles = cv2.HoughCircles(blurred, cv2.HOUGH_GRADIENT, 1, minDist, param1, param2, minRadius, maxRadius) if circles is not None: circles = np.uint16(np.around(circles)) for i in circles[0,:]: cv2.circle(blurred,(i[0], i[1]), i[2], (0, 255, 0), 2)  Maybe Im using wrong function? 

Answer: The main problem in your code is 5th argument to HoughCircles function.  According to documentation the argument list is:  cv2.HoughCircles(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) → circles That means the 5th argument applies circles (it gives an option getting the output by reference, instead of using the returned value).  Because you are not passing circles argument, you must pass named arguments for all arguments after the 4th argument (like param1=param1, param2=param2....). Parameter tuning issues:   Reduce the value of param1. param1 is the higher threshold passed to the Canny. In your case value should be about 30.  Reduce the value of param2 The documentation not so clear, but setting the value around 50 works.  Increase maxRadius value - radius 10 is much smaller than the radius of your circles.   Here is the code:  import numpy as np import cv2 img = cv2.imread('circles.png') gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) blurred = cv2.medianBlur(gray, 25) #cv2.bilateralFilter(gray,10,50,50) minDist = 100 param1 = 30 #500 param2 = 50 #200 #smaller value-> more false circles minRadius = 5 maxRadius = 100 #10 # docstring of HoughCircles: HoughCircles(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) -> circles circles = cv2.HoughCircles(blurred, cv2.HOUGH_GRADIENT, 1, minDist, param1=param1, param2=param2, minRadius=minRadius, maxRadius=maxRadius) if circles is not None: circles = np.uint16(np.around(circles)) for i in circles[0,:]: cv2.circle(img, (i[0], i[1]), i[2], (0, 255, 0), 2) # Show result for testing: cv2.imshow('img', img) cv2.waitKey(0) cv2.destroyAllWindows()   Result:
Question: I'm trying to find a simple way to change the standard output that I get in Jupyter in order to copy and paste tables ouputs in Excel. For example a simple value_counts() give this kind of output in Jupyter lab/notebook: 0 1030971 1 8766 Is there a simple way to mark, copy and paste keeping the format? Even better, could the table have borders? PD. I do not need to export results to excel, I only need to copy a few tables of all my output. Thanks in advance!

Answer: Assuming your DataFrame is named "df", run this line and you'll save the data to your clipboard, letting you paste it where you need to. df.to_clipboard(excel=True)
Question: I have an SQLite database with three columns, and I'm trying to use parameter substitution for tuples to SELECT rows. This is my table: conn = sqlite3.connect("SomeDb.sqlite3") conn.execute(""" CREATE TABLE RoadSegmentDistribution( Source INTEGER, Destination INTEGER, Distribution TEXT ) """)  I know how to substitute with non-tuples, but I cannot figure out how to do it with tuples. Based on this answer, I thought I simply had to substitute each and every value across the list of tuples: for e in conn.execute(""" SELECT * FROM RoadSegmentDistribution WHERE ( Source, Destination ) IN (VALUES (?,?), (?,?), (?,?), (?,?), (?,?)) """, [(1, 2),(2, 3),(4, 5),(6, 7),(8, 9)] ): print(e)  but then I get the error  ProgrammingError: Incorrect number of bindings supplied. The current statement uses 10, and there are 5 supplied.  Obviously this means that I only need one question mark per tuple, right?: for e in conn.execute(""" SELECT * FROM RoadSegmentDistribution WHERE ( Source, Destination ) IN (VALUES (?), (?), (?), (?), (?)) """, [(1, 2),(2, 3),(4, 5),(6, 7),(8, 9)] ): print(e)  But then I get this error:  OperationalError: sub-select returns 1 columns - expected 2  I cannot insert the values manually like in the linked answer, since I don't know what the list parameter contains. This means that I need to do some kind of ",".join() based on the length of the list, but I'll figure that out once I know how to do substitution with a fixed-length list. How would I do this?

Answer: Using the str.join method is indeed a good way to achieve this, given the lack of native support for container-based placeholders in SQL engines:  values = [(1, 2), (2, 3), (4, 5), (6, 7), (8, 9)] for e in conn.execute(f""" SELECT * FROM RoadSegmentDistribution WHERE ( Source, Destination ) IN (VALUES {','.join(f'({",".join("?" * len(t))})' for t in values)}) """, [i for t in values for i in t] ): print(e)  where, with the given values: f""" SELECT * FROM RoadSegmentDistribution WHERE ( Source, Destination ) IN (VALUES {','.join(f'({",".join("?" * len(t))})' for t in values)}) """  would expand into: SELECT * FROM RoadSegmentDistribution WHERE ( Source, Destination ) IN (VALUES (?,?),(?,?),(?,?),(?,?),(?,?))
Question: I'm trying to keep some functions together and tried a Python dataclass for this. I could not come up with or find how to assign a type to a function within a dataclass.  In example below I used a dummy type int, but what I should use correctly instead of int? from dataclasses import dataclass inc = lambda x : x+1 @dataclass class Holder: func: int # but I need a better type signature here h = Holder(inc) assert h.func(1) == 2 

Answer: You should use the Callable type from typing import Callable @dataclass class Holder: func: Callable[[int], int]
Question: I am coding up a dictionary using Django. I want a word to have multiple definitions, if necessary. This would be a one-to-many relationship, but Django does not seem to have a OneToManyField. This is a snippet of my code: class Definition(models.Model): definition = models.CharField(max_length=64) class Word(models.Model): word = models.CharField(max_length=64, unique=True) definitions = models.ForeignKey(Definition, on_delete=models.CASCADE, related_name="word")  I would like to do word.definitions and get back all the definitions of that word. Also, deleting a word should delete all definitions of that word. Finally, a_definition.word should give me the word associated with that definition.

Answer: You have to use ForeignKey in Definition class. Definition will have relation to Word: from django.db import models class Definition(models.Model): definition = models.CharField(max_length=64) word = models.ForeignKey(Word, on_delete=models.CASCADE) class Word(models.Model): word = models.CharField(max_length=64, unique=True)  And you can query it likes this: from .models import Word, Definition word = Word.objects.get(word = 'test') #get Word object definitions = Definition.objects.filter(word = word) #get all Definition objects related to word object above for definition in definitions: #print all definitions related to word print('%s -> %s' % (word.word, definition.definition))