eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have some dataframes as follows: df = pd.DataFrame([[1,2.0],[3,4.0]], index = ['row1','row2'], columns = ['a','b']) df2 = df.iloc[:, :] df3 = df.iloc[:1, :] df4 = df.iloc[:, :1]  Column a is int while column b is float. Question: are df2, df3, df4 view or copy test 1: print(df._is_view, df._is_copy) print(df2._is_view, df2._is_copy) print(df3._is_view, df3._is_copy) print(df4._is_view, df4._is_copy) False None False None False <weakref at 0x7fed1113de90; to 'DataFrame' at 0x7fed11aa80a0> True <weakref at 0x7fed114d65c0; to 'DataFrame' at 0x7fed11aa9ab0>  From this, it says df2, df3 are not a view. But df4 is. Why? test 2: df2.loc['row1', 'b'] = 100.0 print(df1) df3.loc['row1', 'a'] = 1000.0 print(df1) df4.loc['row1', 'a'] = 10000.0 print(df1) a b row1 10 2.0 row2 3 4.0 a b row1 100 2.0 row2 3 4.0 a b row1 100 2.0 row2 3 4.0 /tmp/ipykernel_2006744/1832530048.py:5: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df4.loc['row1', 'a'] = 1000  From this, it can be seen that df's value is updated when df2 or df3 is updated. So df2 and df3 should be a view. Updating df4 does not propagate to df, so df4 seems to be a copy. How come the results are contradicting to _is_view Question 2: The SettingWithCopyWarning when setting df4 says a copy of a slice. What is this refering to? Is "a slice" refering to df4? Then what is the "a copy of a slice" provided I am using .loc?

Answer: You are setting values on a newly created sliced data frame. Don't do it. That's a kind of chained assignment, warned by the document. In your code, the df2 and df3 are views and df4 is a copy. It cannot be determined accurately from the undocumented API _is_view and _is_copy. And 'a copy of a slice' in the warning means the result of df[:, :1] as a copy, where 'a slice' means the symbolic source code df[:, :1] - a Python slicing syntax. In the current Pandas implementation, it cannot be easily defined whether a slice of a data frame is a view or a copy of the original frame, due to the following reasons.  Cell values of a data frame can be stored into multiple NumPy arrays. (See The BlockManager by Uwe for a detail.)  Implementation on tracking references created by slicing is incomplete. (See NDFrame._slice() for example. It does not check if an actual copy was done such as via Block.take_nd().)   So the document says vaguely '... may depend on the context'. The _is_view and _is_copy do not give accurate information. And the internal checking for a chained assignment is not always done. For example, you can see this incompleteness in the following. print('on a heterogenious one') df = pd.DataFrame({'a': [1, 2], 'b': [4, 5], 'c': ['a', 'b']}) df.iloc[:, :1].loc[0, 'a'] = 10 print('on a homogenious one') df = pd.DataFrame({'a': [1, 2], 'b': [4, 5]}) df.iloc[:, :1].loc[0, 'a'] = 10  This outputs the following. on a heterogenious one on a homogenious one test_iloc.py:10: SettingWithCopyWarning: ...
Question: Say I have a generic function that can return a number of different types depending on what properties I select: def json_parser(json_data: Dict[str, Any], property_tree: List[str] ) -> Union[Dict[str, Any], List[str], str, None]: ....  I then call this generic function with specific properties that I know will return a string. def get_language(self, metadata_json: Dict[str, Any]) -> str: return json_parser(metadata_json, ["volumeInfo", "language"])  I get the following error when running Mypy. error: Incompatible return value type (got "Union[Dict[str, Any], List[str], str, None]", expected "List[str]") [return-value]  From my understanding, this is because Mypy doesn't know to expect a string. What is the best way of handling this?

Answer: You've got two options, depending on how careful you want to be. First, typing.cast is a function that takes a type and a value and... magically makes the value have that type. At runtime it's defined as def cast(ty, value): return value  but type-checkers are instructed to treat it as some magic black box. You could write from typing import cast ... return cast(str, json_parser(metadata_json, ["volumeInfo", "language"]))  and your type checker will simply accept the reality that the value is a str. This is not checked at runtime. If you made a mistake and that turned out to be a dictionary or something else, cast will let it pass, and your beautiful statically-typed code will fail with a type error at runtime. That leads us to our second option: assertions. Mypy and other type checkers understand assertions, if statements, and other constructs, and they also understand isinstance. So if you can convince Mypy that you've done your due diligence, it will accept your type outright. result = json_parser(metadata_json, ["volumeInfo", "language"]) assert isinstance(result, str) return result # Accepted as str by mypy  or result = json_parser(metadata_json, ["volumeInfo", "language"]) if isinstance(result, str): return result # Accepted as str by mypy else: raise SomeSpecificError(...)  This is checked at runtime. If someone hands you bad data or you made a mistake, you'll get a runtime error, which is better than nothing.
Question: I am trying to inherit str class for fun. I provided two ways, 1) using super() and 2) using str class in the constructor as follows: class Str2(str): def __init__(self, value): super().__init__() # I did not use `value` here, but my code works! def ishello(self): if self == "Hello": return True else: False s = Str2("Hello") print(s.upper()) # a method of str class print(s.ishello()) # a new method I defined in Str2 class  class Str2(str): def __init__(self, value): str.__init__(value) # It makes sense for me def ishello(self): if self == "Hello": return True else: False s = Str2("Hello") print(s.upper()) # a method of str class print(s.ishello()) # a new method I defined in Str2 class  I am wondering how super().__init__() works in my first code although I did not use value Thanks in advance.

Answer: str.__init__ does not do anything (similar to tuple.__init__, among other immutable classes). The actual initialization happens in __new__. Conceptually, this makes sense, since __new__ returns a new object, while __init__ can be run multiple times on an existing one. That means that whether you call super().__init__ or str.__init__ with or without an argument, or even omit the call entirely, you will get the same result. When you call Str2(value), you are actually calling type.__call__(Str2, value), which passes value to both __new__ and __init__. Since you did not override __new__, you get the full benefit of its behavior no matter what __init__ does. In short, if you don't want to modify value, you can get rid of your custom __init__. If you do want to modify value, override __new__ instead: class Str2(str): def __new__(cls, value): value = "My prefix " + value return super().__new__(cls, value)
Question: Getting error while running the airflow DAG code in visual studio code. Error ImportError: cannot import name 'DummyOperator' from 'airflow.operators' (c:\Users\10679196\AppData\Local\Programs\Python\Python38\lib\site-packages\airflow\operators\__init__.py)  Import Statement from airflow import DAG from airflow.operators import DummyOperator  Version apache-airflow : 2.4.2 python : 3.8.1 pip : 22.3 

Answer: As per documentation the DummyOperator is deprecated and beginning with the version 2.4.0 is not supported any more. You should use from airflow.operators.empty import EmptyOperator  BTW your old import seems also incorrect. For airflow < 2.4.0 this should work: from airflow.operators.dummy import DummyOperator
Question: Getting errors when installing scikit-image with python-3.11.0. The package is simply installed via pip install scikit-image or python -m pip install -U scikit-image. The error messages showed that the problem occur on the wheel building process, and therefore hinder the scikit-image installation. How could I fix this problem? Error messages:  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ INFO: CCompilerOpt.cache_flush[857] : write cache to path -> C:\Users\admin\AppData\Local\Temp\pip-install-z_bg8g6d\scikit-image_ae4333805d744761b97e8cd984f9e2c1\build\temp.win-amd64-3.11\Release\ccompiler_opt_cache_ext.py [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for scikit-image Failed to build scikit-image ERROR: Could not build wheels for scikit-image, which is required to install pyproject.toml-based projects  Environment: affine==2.3.1 attrs==22.1.0 beautifulsoup4==4.11.1 certifi==2022.9.24 charset-normalizer==2.1.1 click==8.1.3 click-plugins==1.1.1 cligj==0.7.2 colorama==0.4.6 contourpy==1.0.6 cycler==0.11.0 docopt==0.6.2 Fiona @ file:///C:/Users/admin/pipwin/Fiona-1.8.21-cp311-cp311-win_amd64.whl fonttools==4.38.0 GDAL @ file:///C:/Users/admin/pipwin/GDAL-3.4.3-cp311-cp311-win_amd64.whl geopandas==0.12.1 idna==3.4 Js2Py==0.74 kiwisolver==1.4.4 matplotlib==3.6.2 munch==2.5.0 numpy==1.23.4 opencv-contrib-python==4.6.0.66 packaging==21.3 pandas==1.5.1 Pillow==9.3.0 pipwin==0.5.2 pyjsparser==2.7.1 pyparsing==3.0.9 PyPrind==2.11.3 pyproj==3.4.0 pySmartDL==1.3.4 python-dateutil==2.8.2 pytz==2022.6 pytz-deprecation-shim==0.1.0.post0 rasterio==1.3.3 requests==2.28.1 rioxarray==0.12.4 Shapely==1.8.5.post1 six==1.16.0 snuggs==1.4.7 soupsieve==2.3.2.post1 tzdata==2022.6 tzlocal==4.2 urllib3==1.26.12 xarray==2022.11.0  Two approaches were suggested:  Downgrades to an older python version: Nathan answer Install Visual Studio C++ compiler, as mentioned in: https://wiki.python.org/moin/WindowsCompilers  However, I would like to stay on current python version, and I'm getting stuck in the second approach.

Answer: In my case, this problem was solved via installing a wheel file from: https://www.lfd.uci.edu/~gohlke/pythonlibs/#scikit-image In my case, I download the cp311 windows amd64 version.  Then, install the .whl file to the virtual environment (env) D:\env>python -m pip install D:\Download\scikit_image-0.19.3-cp311-cp311-win_amd64.whl  Finally, the scikit-image was successfully installed.
Question: I have a image with resolution 816x624, and would need to make a 640x640 image out of it. To do so, i'd have to crop the long side (centered), and fill up the short side with black (centered). the resulting image should be the center of the starting image, with a small black strip on top and bottom. How can this be done? i tried with:  crop_img = img1[int(h1 / 2 - 640 / 2):int(h1 / 2 + 640 / 2), int(w1 / 2 - 640 / 2):int(w1 / 2 + 640/ 2)]  but this does not work because h1 i ssmaller than 640. 

Answer: Given an imput image img, an expected height h and an expected width w: def resize_img(img, h, w): #cut the image cutted_img = img[ max(0, int(img.shape[0]/2-h/2)):min(img.shape[0], int(img.shape[0]/2+h/2)), max(0, int(img.shape[1]/2-w/2)):min(img.shape[1], int(img.shape[1]/2+w/2)), ] #pad the image padded_img = np.zeros(shape=(h,w,3), dtype=img.dtype) padded_img[ int(padded_img.shape[0]/2-cutted_img.shape[0]/2):int(padded_img.shape[0]/2+cutted_img.shape[0]/2), int(padded_img.shape[1]/2-cutted_img.shape[1]/2):int(padded_img.shape[1]/2+cutted_img.shape[1]/2), ] = cutted_img return padded_img  some exaples: url = "https://www.planetware.com/wpimages/2020/02/france-in-pictures-beautiful-places-to-photograph-eiffel-tower.jpg" response = requests.get(url) start_img = np.array(PIL.Image.open(BytesIO(response.content))) plt.imshow(start_img) plt.show() #shape = (487, 730, 3)   plt.imshow(resize_img(start_img, 640, 640)) plt.show() #shape = (640, 640, 3)   I tested on some other pictures: shape = (1020, 680, 3)   shape = (450, 254, 3)   shape = (847, 564, 3)   All resized images have size (640, 640, 3) and seem to be properly padded. PS. I have used the following libraries: import numpy as np from matplotlib import pyplot as plt import PIL import requests from io import BytesIO
Question: Problem: I have a ndarray (2000,7) and I want to count the numbers of Nan's per column and save it in an ndarray Tried: number_nan_in_arr = np.count_nonzero(np.isnan(arr))  But this count the total number of nan's over all columns Solution: ?

Answer: You can add axis=0 to count null per column: number_nan_in_arr = np.count_nonzero(np.isnan(arr), axis=0)  Example: import numpy as np a = np.array([[0, np.nan, 7, 0], [3, 0, 2, np.nan]]) np.count_nonzero(np.isnan(a), axis=0)  Output:  array([0, 1, 0, 1], dtype=int64)
Question: Consider the following code block: class MyException(Exception): __hash__ = None try: raise ExceptionGroup("Foo", [ MyException("Bar") ]) except* Exception: pass  The except* should catch any number of exceptions of any kind, thrown together as an ExceptionGroup (or a single exception of any kind if thrown alone, come to that). Instead, an unhandled TypeError occurs during handling of our ExceptionGroup, allegedly at "line -1" of our module:  + Exception Group Traceback (most recent call last): | File "C:\Users\Josep\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_62.py", line 6, in <module> | raise ExceptionGroup("Foo", [ | ExceptionGroup: Foo (1 sub-exception) +-+---------------- 1 ---------------- | MyException: Bar +------------------------------------ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Josep\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_62.py", line -1, in <module> TypeError: unhashable type: 'MyException'  (If we replace the except* Exception with something more specific, say except* ValueError or except* MyException, the same thing happens. If we try to just raise a single MyException and catch it normally with except MyException, that works fine.) Normal except clauses don't care whether exceptions are hashable. I could not find this quirk of except* documented in PEP-654 or the Python 3.11 release notes. Is this intended behavior, or is it simply a bug in the Python implementation I'm using? (For those who want to reproduce this behavior, I'm using Python 3.11.0, 64-bit, on Windows.)

Answer: This was reported at https://github.com/python/cpython/issues/99181 and we have a PR to fix it. Should be fixed in 3.11.1.
Question: I have a Dask process that triggers 100 workers with a map function: worker_args = .... # array with 100 elements with worker parameters futures = client.map(function_in_worker, worker_args) worker_responses = client.gather(futures)  I use docker where each worker is a container. I have configured docker to spawn 20 workers/containers, like so: docker-compose up -d --scale worker=20  The problem is that my machine crashes because the map function triggers 20 workers in parallel and that makes memory and CPU exceed the maximum. I want to keep the configuration of 20 workers because I use the workers for other functions that don't require large amount of memory. How to limit the map function to, say, 5 workers in parallel?

Answer: dask does not dynamically adjust worker resources depending on how many workers are idle. In the example you provided, once 20 workers are initiated, if only 5 workers are used, then they will not be allocated the resources from the remaining 15 workers that are idle. If that's acceptable (e.g. because the idle resources are being utilized by an external program), then one way to restrict work to 5 workers is to explicitly specify them via workers kwarg to .map call: # instantiate workers from distributed import Client c = Client(n_workers=20) # select at most 5 workers from the available list selected_workers = list(c.scheduler_info()['workers'])[:5] dummy_function = lambda x: x**2 futs = c.map(dummy_function, range(10), workers=selected_workers)  Another way to control workload allocation is to use resources kwarg, see these related answers: 0, 1, 2, 3.
Question: In the UDP client example in the Python docs, they use loop.create_future() to create a new Future. The main program awaits this future until result is set on it, at which point the program cleans up resources and terminates. However, I have always used an asyncio.Event for this kind of thing. Is there any difference between these two techniques? Is there any reason to prefer the Future instead of the Event? loop = asyncio.get_running_loop() future = loop.create_future() await future event = asyncio.Event() await event.wait() 

Answer: They can be both used for synchronization, but a Future has a proper result and can raise exceptions. So, Event provides less features, but when the use case is only synchronization, it may express the intent better and be less error-prone. In fact, an Event is implemented as a list of futures.
Question: I am attempting to understand what this snippet of code does: passwd1=re.sub(r'^.*? --', ' -- ', line) password=passwd1[4:]  I understand that the top line uses regex to remove the " -- ", and the bottom line I think removes something as well? I went back to this code after a while and need to improve it but to do that I need to understand this again. I've been trying to read regex docs to no avail, what is this: r'^.*?  at the beginning of the regex?.

Answer: To break r'^.*? -- into pieces:  r in front of a string in Python lets the interpreter know that it's a regex string. This lets you not have to do a bunch of confusing character escaping. The ^ tells the regex to match only from the beginning of the string. .*? tells the regex to match any number of characters up to... --, which is a literal match.  The sum of this is that it will match any string, starting at the beginning of a line up to the -- demarcation. Since it is re.sub(), the matched part of the string will be replaced with --. This is why something like Google -- MyPassword becomes  -- MyPassword. The second line is a simple string slice, dropping the first four elements (characters) of the string. This might be superfluous - you could just substitute the match with an empty string like this: passwd1 = re.sub(r'^.* --', '', line)  This achieves the same result. Note I've dropped the ?, which is also superfluous here, because the * has a similar but broader effect. There are some technical differences, but I don't think you need it for your stated purpose. ? will match zero or one of the previous character - in this case a ., which is 'any character'. The * will match zero or more of the previous character. .* is what is known as a greedy quantifier, and .*? a lazy quantifier. That is, the greedy quantifier will match as much as possible, and the lazy will match as little as possible. The difference between ^.*? -- and ^.* --  is what is matched in this case: Something something -- mypassword -- yourpassword  In the greedy case, the first two clauses ('something something -- mypassword') are matched and deleted. In the lazy case, only 'something something' is deleted. Most passwords don't include spaces, nevermind ' -- ', so you probably want to use the greedy version.
Question: The solutions shown in How to change the figure size of a seaborn axes or figure level plot do not work for seaborn.objects. This question is about the new interface added in seaborn v0.12. Tried different ways to set the plotted image size but no one worked, below is the code, how to set the below image height and width by pixel or inch? About seaborn.objects.Plot.add: import pandas as pd from seaborn import objects as so df = sns.load_dataset('planets') p = so.Plot(data=df, x='orbital_period', y='mass').add(so.Dot()) p.save('output.png') p.show()  

Answer: With Plot.layout: ( so.Plot(data=d, x="point_x", y="point_y") .add(so.Dot()) .layout(size=(w, h)) # in inches * (dpi / 100) .save("output.png", dpi=dpi) # e.g. dpi=100 .show() )
Question: I am trying to use scrapy and playwright to scrape dynamic webpages, I installed scrapy and playwright, however, when I try to run my spider, i get this error. ImportError: cannot import name 'PageCoroutine' from 'scrapy_playwright.page' (C:\Ali\DataCamp\Web Scraping in Python\Scrapy\venv\lib\site-packages\scrapy_playwright\page.py) This is my code(it's a test code): import scrapy from scrapy_playwright.page import PageCoroutine class PwspiderSpider(scrapy.Spider): name = 'pwspider' def start_requests(self): yield scrapy.Request("https://shoppable-campaign-demo.netlify.app/#/", meta=dict(playwright=True, playwright_include_page=True, playwright_page_coroutine=[PageCoroutine('wait_for_selector', 'div#productListing')])) async def parse(self, response): yield {'text': response.text}  I even added the DOWNLOAD_HANDLERS and the TWISTED_REACTOR in the settings file.

Answer: PageCoroutine is deprecated/obsolute. Use playwright_page_methods instead. Working code as an example: import scrapy from scrapy_playwright.page import PageMethod class TestSpider(scrapy.Spider): name = "test" def start_requests(self): yield scrapy.Request( url="https://shoppable-campaign-demo.netlify.app/#/", callback=self.parse, meta={ "playwright": True, "playwright_page_methods": [ PageMethod("wait_for_selector", '.card-body'), ], }, ) def parse(self, response): products = response.xpath('//*[@class="card-body"]') for product in products: yield { 'title':product.xpath('.//*[@class="card-title"]/text()').get() }  Output: {'title': 'Oxford Loafers'} 2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/> {'title': 'Ankle-length Slack'} 2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/> {'title': 'White Baseball Cap'} 2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/> {'title': 'Triangle Bikini Top'} 2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/> {'title': 'Short Blazer'} 2022-11-05 20:40:40 [scrapy.core.engine] INFO: Closing spider (finished) 2022-11-05 20:40:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 235, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 39851, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'elapsed_time_seconds': 41.370211, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2022, 11, 5, 14, 40, 40, 261151), 'item_scraped_count': 5,
Question: I am a total beginner to Python and recently had a question while experimenting with lists. I have a for loop that increases a variable 'x' and generates a random number every time. I want to add this random number to a list, but when I try assigning the index value of the random number to x, I get this error message: Traceback (most recent call last): File "<stdin>", line 6, in <module> IndexError: list assignment index out of range  Any help would be greatly appreciated! My code: import random as number x = 0 values = [] while x < 9: values[x] = number.randint(1,91) x += 1 print(values) 

Answer: import random as number values = [number.randint(1,91) for _ in range(9)]
Question: I'm trying to get back to Python but I don't get why the following code doesn't work as intended. class Cat: age = 0 class Dog(Cat): pass Dog.age = 1 Cat.age = 2 print(Dog.age, Cat.age)  My output is: 1 2  But why doesn't Dog.age equals 2? Dog is a subclass of Cat and modifying the class variable of the superclass Cat would normally affect every subclass that inherits the variable as well.

Answer: Any property of Dog will override a property inherited from Cat. You can re-define a value in Cat, but it won't matter because it has already been overridden by the child. For example: class Cat: age = 0 # Cat.age = 0 class Dog(Cat): pass # Dog.age = Cat.age = 0 Dog.age=1 # Dog.age = 1, and Dog.age no longer points to Cat.age Cat.age=2 # Cat.age = 2 print(Dog.age, Cat.age) # Dog.age is no longer Cat.age. They are completely different  Contrast that with this: class Cat: age = 0 # Cat.age = 0 class Dog(Cat): pass # Dog.age = Cat.age = 0 Cat.age = 10 # Cat.age = 10 print(Dog.age, Cat.age) # Dog.age points to Cat.age, so Dog.age resolves to 10