eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have installed the AWS CLI using pip on my Python 3.9.0a1 alpine Docker image. Installation went fine. When I run the aws command, I'm getting the following error. aws Traceback (most recent call last): File "/usr/local/bin/aws", line 27, in <module> sys.exit(main()) File "/usr/local/bin/aws", line 23, in main return awscli.clidriver.main() File "/usr/local/lib/python3.9/site-packages/awscli/clidriver.py", line 68, in main driver = create_clidriver() File "/usr/local/lib/python3.9/site-packages/awscli/clidriver.py", line 77, in create_clidriver load_plugins(session.full_config.get('plugins', {}), File "/usr/local/lib/python3.9/site-packages/awscli/plugin.py", line 44, in load_plugins modules = _import_plugins(plugin_mapping) File "/usr/local/lib/python3.9/site-packages/awscli/plugin.py", line 61, in _import_plugins module = __import__(path, fromlist=[module]) File "/usr/local/lib/python3.9/site-packages/awscli/handlers.py", line 42, in <module> from awscli.customizations.history import register_history_mode File "/usr/local/lib/python3.9/site-packages/awscli/customizations/history/__init__.py", line 24, in <module> from awscli.customizations.history.db import DatabaseConnection File "/usr/local/lib/python3.9/site-packages/awscli/customizations/history/db.py", line 19, in <module> from collections import MutableMapping ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/local/lib/python3.9/collections/__init__.py) python --version Python 3.9.0a1  Do I need to install any other module to fix this error message?

Answer: collections.MutableMapping has been deprecated since Python 3.3, and was officially removed since Python 3.9. Excerpt from the documentation:  Deprecated since version 3.3, will be removed in version 3.9: Moved Collections Abstract Base Classes to the collections.abc module.  You can either wait for a Python 3.9-compatible version of awscli to be released, or patch the aws script (under your /usr/local/bin) yourself like this for the time being: ... import collections from collections import abc collections.MutableMapping = abc.MutableMapping import awscli.clidriver
Question: I have the following image which is a receipt image and a lot of white space around the receipt in focus. I would like to crop the white space. I can't manually crop it so I'm looking for a way that I could do it.  Cropped one:  Tried this code from the following post: How to remove whitespace from an image in OpenCV? gray = load_image(IMG_FILE) # image file gray = 255*(gray < 128).astype(np.uint8) coords = cv2.findNonZero(gray) # Find all non-zero points (text) x, y, w, h = cv2.boundingRect(coords) # Find minimum spanning bounding box rect = load_image(IMG_FILE)[y:y+h, x:x+w] # Crop the image - note we do this on the original image  it's cropping a tiny part of the white space.

Answer: Here's a simple approach:  Obtain binary image. Load the image, convert to grayscale, apply a large Gaussian blur, and then Otsu's threshold Perform morphological operations. We first morph open with a small kernel to remove noise then morph close with a large kernel to combine the contours Find enclosing bounding box and crop ROI. We find the coordinates of all non-zero points, find the bounding rectangle, and crop the ROI.   Here's the detected ROI to crop highlighted in green  Cropped ROI  import cv2 # Load image, grayscale, Gaussian blur, Otsu's threshold image = cv2.imread('1.jpg') original = image.copy() gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) blur = cv2.GaussianBlur(gray, (25,25), 0) thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] # Perform morph operations, first open to remove noise, then close to combine noise_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3)) opening = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, noise_kernel, iterations=2) close_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (7,7)) close = cv2.morphologyEx(opening, cv2.MORPH_CLOSE, close_kernel, iterations=3) # Find enclosing boundingbox and crop ROI coords = cv2.findNonZero(close) x,y,w,h = cv2.boundingRect(coords) cv2.rectangle(image, (x, y), (x + w, y + h), (36,255,12), 2) crop = original[y:y+h, x:x+w] cv2.imshow('thresh', thresh) cv2.imshow('close', close) cv2.imshow('image', image) cv2.imshow('crop', crop) cv2.waitKey()
Question: I've a dataframe with list of items separated by , commas as below. +----------------------+ | Items | +----------------------+ | X1,Y1,Z1 | | X2,Z3 | | X3 | | X1,X2 | | Y2,Y4,Z2,Y5,Z3 | | X2,X3,Y1,Y2,Z2,Z4,X1 | +----------------------+  Also I've 3 list of arrays which has all items said above clubbed into specific groups as below X = [X1,X2,X3,X4,X5] Y = [Y1,Y2,Y3,Y4,Y5] Z = [Z1,Z2,Z3,Z4,Z5] my task is to split the each value in the dataframe & check individual items in the 3 arrays and if an item is in any of the array, then it should concatenate the name of the groups which it is found, separated with &. Also if many items are in the same group/array, then it should mention the number of occurrence as well. My desired output is as below. refer Category column +----------------------+--------------+ | Items | Category | +----------------------+--------------+ | X1,Y1,Z1 | X & Y & Z | | X2,Z3 | X & Z | | X3 | X | | X1,X2 | 2X | | Y2,Y4,Z2,Y5,Z3 | 3Y & 2Z | | X2,X3,Y1,Y2,Z2,Z4,X1 | 3X & 2Y & 2Z | +----------------------+--------------+  X,Y, and Z are the name of the arrays. how shall I start to solve this using pandas? please guide.

Answer: Setup df = pd.DataFrame( [['X1,Y1,Z1'], ['X2,Z3'], ['X3'], ['X1,X2'], ['Y2,Y4,Z2,Y5,Z3'], ['X2,X3,Y1,Y2,Z2,Z4,X1']], columns=['Items'] ) X = ['X1', 'X2', 'X3', 'X4', 'X5'] Y = ['Y1', 'Y2', 'Y3', 'Y4', 'Y5'] Z = ['Z1', 'Z2', 'Z3', 'Z4', 'Z5']   Counter from collections import Counter M = {**dict.fromkeys(X, 'X'), **dict.fromkeys(Y, 'Y'), **dict.fromkeys(Z, 'Z')} num = lambda x: {1: ''}.get(x, x) cat = ' & '.join fmt = lambda c: cat(f'{num(v)}{k}' for k, v in c.items()) cnt = lambda x: Counter(map(M.get, x.split(','))) df.assign(Category=[*map(fmt, map(cnt, df.Items))]) Items Category 0 X1,Y1,Z1 X & Y & Z 1 X2,Z3 X & Z 2 X3 X 3 X1,X2 2X 4 Y2,Y4,Z2,Y5,Z3 3Y & 2Z 5 X2,X3,Y1,Y2,Z2,Z4,X1 3X & 2Y & 2Z   OLD STUFF pandas.Series.str.get_dummies and groupby First convert the definitions of X, Y, and Z into one dictionary, then use that as the argument for groupby on axis=1 M = {**dict.fromkeys(X, 'X'), **dict.fromkeys(Y, 'Y'), **dict.fromkeys(Z, 'Z')} counts = df.Items.str.get_dummies(',').groupby(M, axis=1).sum() counts X Y Z 0 1 1 1 1 1 0 1 2 1 0 0 3 2 0 0 4 0 3 2 5 3 2 2  Add the desired column Work in Progress I don't like this solution def fmt(row): a = [f'{"" if v == 1 else v}{k}' for k, v in row.items() if v > 0] return ' & '.join(a) df.assign(Category=counts.apply(fmt, axis=1)) Items Category 0 X1,Y1,Z1 X & Y & Z 1 X2,Z3 X & Z 2 X3 X 3 X1,X2 2X 4 Y2,Y4,Z2,Y5,Z3 3Y & 2Z 5 X2,X3,Y1,Y2,Z2,Z4,X1 3X & 2Y & 2Z   NOT TO BE TAKEN SERIOUSLY Because I'm leveraging the character of your contrived example and there is nowai you should depend on the first character of your values to be the thing that differentiates them. from operator import itemgetter df.Items.str.get_dummies(',').groupby(itemgetter(0), axis=1).sum() X Y Z 0 1 1 1 1 1 0 1 2 1 0 0 3 2 0 0 4 0 3 2 5 3 2 2
Question: I have something similar to this: from typing import Type class Foo: pass def make_a_foobar_class(foo_class: Type[Foo]) -> Type[Foo]: class FooBar(foo_class): # this.py:10: error: Variable "foo_class" is not valid as a type # this.py:10: error: Invalid base class "foo_class" pass return FooBar print(make_a_foobar_class(Foo)())  Running mypy throws these two errors (added as comments ^) at line class FooBar(foo_class): The code seems to work just fine: $ python this.py <__main__.make_a_foobar_class.<locals>.FooBar object at 0x10a422be0>  What am I doing wrong?

Answer: Mypy, and the PEP 484 ecosystem in general, does not support creating classes with dynamic base types. This is likely because supporting such a feature is not worth the additional complexity: the type checker would need to implement additional logic/additional passes since it can no longer cleanly determine what exactly the parent type is by just examining the set of variable names that are currently in scope, and can also no longer accurately type check code using the new dynamic class in the general case. In any case, I would recommend either redesigning your code to avoid doing this, perhaps by using composition over inheritance or something. Alternatively, you can suppress the errors mypy is generating by adding a # type: ignore annotation. This annotation will filter out all errors associated with that particular line once type checking is done. For example: from typing import Type class Foo: pass def make_a_foobar_class(foo_class: Type[Foo]) -> Type[Foo]: class FooBar(foo_class): # type: ignore pass return FooBar print(make_a_foobar_class(Foo)())
Question: The zoom in, zoom out and zoom to percentage buttons are disabled in Plots pane in Spyder as shown here. Any idea how to enable them? The should be enabled as seen here.  Specs Spyder version: 4.0.0 OS: elementary OS Python: 3.7.5 64-bit Kernel: Linux 5.4.7-050407-generic Laptop: Thinkpad E585 Failed attempts Following changes in preferences didn't help: 1. Changing Backend from inline to automatic. 2. Changing Format from png to svg. 3. Changing Resolution from 72 to 150 dpi.

Answer: (Spyder maintainer here) To be able to use those buttons you need to deactivate the option called Fit plots to window, present in the Options menu of the Plots pane:
Question: I'm trying to split a column using regex, but can't seem to get the split correctly. I'm trying to take all the trailing CAPS and move them into a separate column. So I'm getting all the CAPS that are either 2-4 CAPS in a row. However, it's only leaving the 'Name' column while the 'Team' column is blank. Here's my code: import pandas as pd url = "https://www.espn.com/nba/stats/player/_/table/offensive/sort/avgAssists/dir/desc" df = pd.read_html(url)[0].join(pd.read_html(url)[1]) df[['Name','Team']] = df['Name'].str.split('[A-Z]{2,4}', expand=True)  I want this: print(df.head(5).to_string()) RK Name POS GP MIN PTS FGM FGA FG% 3PM 3PA 3P% FTM FTA FT% REB AST STL BLK TO DD2 TD3 PER 0 1 LeBron JamesLA SF 35 35.1 24.9 9.6 19.7 48.6 2.0 6.0 33.8 3.7 5.5 67.7 7.9 11.0 1.3 0.5 3.7 28 9 26.10 1 2 Ricky RubioPHX PG 30 32.0 13.6 4.9 11.9 41.3 1.2 3.7 31.8 2.6 3.1 83.7 4.6 9.3 1.3 0.2 2.5 12 1 16.40 2 3 Luka DoncicDAL SF 32 32.8 29.7 9.6 20.2 47.5 3.1 9.4 33.1 7.3 9.1 80.5 9.7 8.9 1.2 0.2 4.2 22 11 31.74 3 4 Ben SimmonsPHIL PG 36 35.4 14.9 6.1 10.8 56.3 0.1 0.1 40.0 2.7 4.6 59.0 7.5 8.6 2.2 0.7 3.6 19 3 19.49 4 5 Trae YoungATL PG 34 35.1 28.9 9.3 20.8 44.8 3.5 9.4 37.5 6.7 7.9 85.0 4.3 8.4 1.2 0.1 4.8 11 1 23.47  to become this: print(df.head(5).to_string()) RK Name Team POS GP MIN PTS FGM FGA FG% 3PM 3PA 3P% FTM FTA FT% REB AST STL BLK TO DD2 TD3 PER 0 1 LeBron James LA SF 35 35.1 24.9 9.6 19.7 48.6 2.0 6.0 33.8 3.7 5.5 67.7 7.9 11.0 1.3 0.5 3.7 28 9 26.10 1 2 Ricky Rubio PHX PG 30 32.0 13.6 4.9 11.9 41.3 1.2 3.7 31.8 2.6 3.1 83.7 4.6 9.3 1.3 0.2 2.5 12 1 16.40 2 3 Luka Doncic DAL SF 32 32.8 29.7 9.6 20.2 47.5 3.1 9.4 33.1 7.3 9.1 80.5 9.7 8.9 1.2 0.2 4.2 22 11 31.74 3 4 Ben Simmons PHIL PG 36 35.4 14.9 6.1 10.8 56.3 0.1 0.1 40.0 2.7 4.6 59.0 7.5 8.6 2.2 0.7 3.6 19 3 19.49 4 5 Trae Young ATL PG 34 35.1 28.9 9.3 20.8 44.8 3.5 9.4 37.5 6.7 7.9 85.0 4.3 8.4 1.2 0.1 4.8 11 1 23.47 

Answer: You may extract the data into two columns by using a regex like ^(.*?)([A-Z]+)$ or ^(.*[^A-Z])([A-Z]+)$: df[['Name','Team']] = df['Name'].str.extract('^(.*?)([A-Z]+)$', expand=True)  This will keep all up to the last char that is not an uppercase letter in Group "Name" and the last uppercase letters in Group "Team". See regex demo #1 and regex demo #2 Details  ^ - start of a string (.*?) - Capturing group 1: any zero or more chars other than line break chars, as few as possible or (.*[^A-Z]) - any zero or more chars other than line break chars, as many as possible, up to the last char that is not an ASCII uppercase letter (granted the subsequent patterns match) (note that this pattern implies there is at least 1 char before the last uppercase letters) ([A-Z]+) - Capturing group 2: one or more ASCII uppercase letters $ - end of string.
Question: I am trying to install flair. It is throwing below error when executing below command: pip install flair ERROR: Packages installed from PyPI cannot depend on packages which are not also hosted on PyPI. tiny-tokenizer depends on SudachiDict_core@ https://object-storage.tyo2.conoha.io/v1/nc_2520839e1f9641b08211a5c85243124a/sudachi/SudachiDict_core-20190927.tar.gz I thought installing this package explicitly might fix the error but it doesn't. The error remains same. The installed version of SudachiDict-core is below: SudachiDict-core 0.0.0 Below is the Environment:  OS: Windows 10  Python: 3.6 (64 bit)  Any hint is appreciated. Thank you! Note:   First hurdle when installing flair was torch package. It was resolved simply when torch package is installed. The error looked like below: ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from flair) (from verERROR: No matching distribution found for torch>=1.1.0 (from flair) 

Answer: It is strange running below command solved the problem. pip install flair==0.4.3 I assume that the problem is in a latest version 0.4.4 (and its dependencies).  Note: I had torch==1.1.0 package already installed.
Question: In a Python regular expression, I encounter this singular problem. Could you give instruction on the differences between re.findall('(ab|cd)', string) and re.findall('(ab|cd)+', string)? import re string = 'abcdla' result = re.findall('(ab|cd)', string) result2 = re.findall('(ab|cd)+', string) print(result) print(result2)  Actual Output is: ['ab', 'cd'] ['cd']  I'm confused as to why does the second result doesn't contain 'ab' as well?

Answer: I don't know if this will clear things more, but let's try to imagine what happen under the hood in a simple way, we going to sumilate what happen using match  # group(0) return the matched string the captured groups are returned in groups or you can access them # using group(1), group(2)....... in your case there is only one group, one group will capture only # one part so when you do this string = 'abcdla' print(re.match('(ab|cd)', string).group(0)) # only 'ab' is matched and the group will capture 'ab' print(re.match('(ab|cd)+', string).group(0)) # this will match 'abcd' the group will capture only this part 'cd' the last iteration  findall match and consume the string at the same time let's imagine what happen with this REGEX '(ab|cd)':  'abcdabla' ---> 1: match: 'ab' | capture : ab | left to process: 'cdabla' 'cdabla' ---> 2: match: 'cd' | capture : cd | left to process: 'abla' 'abla' ---> 3: match: 'ab' | capture : ab | left to process: 'la' 'la' ---> 4: match: '' | capture : None | left to process: '' --- final : result captured ['ab', 'cd', 'ab']  Now the same thing with '(ab|cd)+'  'abcdabla' ---> 1: match: 'abcdab' | capture : 'ab' | left to process: 'la' 'la' ---> 2: match: '' | capture : None | left to process: '' ---> final result : ['ab']  I hope this clears thing a little bit.
Question: I have the following tox.ini file: [tox] envlist = flake8,py{35,36,37,38}{,-keyring} [testenv] usedevelop = True install_command = pip install -U {opts} {packages} deps = .[test] keyring: .[keyring] setenv = COVERAGE_FILE = .coverage.{envname} commands= pytest {toxinidir}/tests -n 4 {posargs} [testenv:flake8] basepython = python3 deps = flake8 commands= flake8 src tests [flake8] ignore: F401,E402,E501,W605,W503  When I run the tox command, it creates a .tox folder containing a folder for every environment specified in the [tox] section of the tox.ini. I would like to automatically get rid of these particular folders after the test have succeeded when running tox without having to manually run rm -rf .tox/NAME_OF_THE_ENV. I have searched through the tox documentation but I have found nothing. Is it possible to do so? If yes, how?

Answer: I found a way by creating a tox hook. This hook runs the shutil.rmtree command after the tests have been run inside the env. In a tox_clean_env.py file: import shutil from tox import hookimpl @hookimpl def tox_runtest_post(venv): try: shutil.rmtree(venv.path) except Exception as e: print("An exception occurred while removing '{}':".format(venv.path)) print(e)  I created a package around this code and I just need to install it using pip. In my setup.py, in the setup function:  entry_points={"tox": ["clean_env = tox_clean_env"]},
Question: I have a PDF file of around 7000 pages and 479 MB. I have create a python script using PyPDF4 to extract only specific pages if the pages contain specific words. The script works but the new PDF file, even though it has only 650 pages from the original 7000, now has more MB that the original file (498 MB to be exactly).  Is there any way to lower the filesize of the new PDF? The script I used: from PyPDF4 import PdfFileWriter, PdfFileReader import os import re output = PdfFileWriter() input = PdfFileReader(open('Binder.pdf', 'rb')) # open input for i in range(0, input.getNumPages()): content = "" content += input.getPage(i).extractText() + "\n" #Format 1 RS = re.search('FIGURE', content) RS1 = #... Only one search given as example. I have more, but are irrelevant for the question. #.... # Format 2 RS20 = re.search('FIG.', content) RS21 = #... Only one search given as example. I have more, but are irrelevant for the question. #.... if (all(v is not None for v in [RS, RS1, RS2, RS3, RS4, RS5, RS6, RS7, RS8, RS9]) or all(v is not None for v in [RS20, RS21, RS22, RS23, RS24, RS25, RS26, RS27, RS28, RS29, RS30, RS30])): p = input.getPage(i) output.addPage(p) #Save pages to new PDF file with open('ExtractedPages.pdf', 'wb') as f: output.write(f) 

Answer: After a lot of searching found some solutions. The only problem with the exported PDF file was that it was uncompressed. So I needed a solution to compress a PDF file:  PyPDF2 and/or PyPDF4 do not have an option to compress PDFs. PyPDF2 had the compressContentStreams() method, which doesn't work. Found a few other solutions that claim to compress PDFs, but none worked for me (adding them here just in case they work for others): pylovepdf ; pdfsizeopt ; pdfc The first solution that worked for me was Adobe Acrobat professional. It reduced the size from 498 MB to 2.99 MB. [Best Solution] As an alternative, open source solution that works, I found coherentpdf. For Windows you can download the pre-built PDF squeezer tool. Then in cmd: cpdfsqueeze.exe input.pdf output.pdf  This actually compressed the PDF even more than Adobe Acrobat. From 498 MB to 2.48 MB. Compressed to 0.5% from original. I think this is the best solution as it can be added to your Python Code.  Edit: Found another Free solution that has a GUI also. PDFsam. You can use the Merge feature on one PDF file, and in the advanced Settings make sure you have the Compress Output checked. This compressed from 498 to 3.2 MB.
Question: I understand what register_buffer does and the difference between register_buffer and register_parameters.  But what is the precise definition of a buffer in PyTorch?

Answer: This can be answered looking at the implementation: def register_buffer(self, name, tensor): if '_buffers' not in self.__dict__: raise AttributeError( "cannot assign buffer before Module.__init__() call") elif not isinstance(name, torch._six.string_classes): raise TypeError("buffer name should be a string. " "Got {}".format(torch.typename(name))) elif '.' in name: raise KeyError("buffer name can't contain \".\"") elif name == '': raise KeyError("buffer name can't be empty string \"\"") elif hasattr(self, name) and name not in self._buffers: raise KeyError("attribute '{}' already exists".format(name)) elif tensor is not None and not isinstance(tensor, torch.Tensor): raise TypeError("cannot assign '{}' object to buffer '{}' " "(torch Tensor or None required)" .format(torch.typename(tensor), name)) else: self._buffers[name] = tensor  That is, the buffer's name:  must be a string: not isinstance(name, torch._six.string_classes) cannot contain a . (dot): '.' in name cannot be an empty string: name == '' cannot be an attribute of the Module: hasattr(self, name) should be unique: name not in self._buffers  and the tensor (guess what?):  should be a Tensor: isinstance(tensor, torch.Tensor)  So, the buffer is just a tensor with these properties, registered in the _buffers attribute of a Module;
Question: I'm using the official python (2.7) client. I want to define a metric with some labels but I don't always have them all the labels to send. When I send only some of them I get the error:  AttributeError: 'Counter' object has no attribute '_value'  This is the code I used: c = Counter("counterTest, "explain this counter, labelnames=("label1", "label2",), namespace="namespace") c.labels(label1="1").inc(1)  Is this a limitation in the python library? Or maybe it's a limitation on the Prometheus end?

Answer: You must always specify all labels, how else would we know which series it is that you want to increment? You can specify an empty string as a label value, though this may cause confusion for your users.
Question: One can share Python as a source distribution (.tar.gz format) or as a built distribution (wheels format). As I understand it, the point of built distributions is:  Save time: Compilation might be pretty time-consuming. We can do this once on the server and share it for many users. Reduce requirements: The user does not have to have a compiler installed  However, those two arguments for bdist files seem not to hold for pure-python packages. Still, I see that natsort comes in both, a sdist and a bdist. Is there any advantage of sharing a pure-python package in bdist format?

Answer: From pythonwheels.com:  Advantages of wheels  Faster installation for pure Python and native C extension packages. Avoids arbitrary code execution for installation. (Avoids setup.py) Installation of a C extension does not require a compiler on Linux, Windows or macOS. Allows better caching for testing and continuous integration. Creates .pyc files as part of installation to ensure they match the Python interpreter used. More consistent installs across platforms and machines.   So for me, I think the first and second points are most meaningful for a pure Python package. It's smaller, faster and also more secure.
Question: In Python, exceptions are classes and cased as such. For example: OSError. However, there are some exceptions, such as those in the socket module, that are named in lower-case. For example: socket.timeout, socket.error. Why is this?

Answer: According to the docs,  exception socket.error A deprecated alias of OSError. Changed in version 3.3: Following PEP 3151, this class was made an alias of OSError.  PEP 3151 says  while standard exception types live in the root namespace, they are visually distinguished by the fact that they use the CamelCase convention, while almost all other builtins use lowercase naming (except True, False, None, Ellipsis and NotImplemented)
Question: Is there a null propagation operator ("null-aware member access" operator) in Python so I could write something like var = object?.children?.grandchildren?.property  as in C#, VB.NET and TypeScript, instead of var = None if not myobject\ or not myobject.children\ or not myobject.children.grandchildren\ else myobject.children.grandchildren.property 

Answer: No. There is a PEP proposing the addition of such operators but it has not (yet) been accepted. In particular, one of the operators proposed in PEP 505 is  The "None-aware attribute access" operator ?. ("maybe dot") evaluates the complete expression if the left hand side evaluates to a value that is not None