eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: From a segmentation mask, I am trying to retrieve what labels are being represented in the mask.  This is the image I am running through a semantic segmentation model in AWS Sagemaker.  Code for making prediction and displaying mask. from sagemaker.predictor import json_serializer, json_deserializer, RealTimePredictor from sagemaker.content_types import CONTENT_TYPE_CSV, CONTENT_TYPE_JSON %%time ss_predict = sagemaker.RealTimePredictor(endpoint=ss_model.endpoint_name, sagemaker_session=sess, content_type = 'image/jpeg', accept = 'image/png') return_img = ss_predict.predict(img) from PIL import Image import numpy as np import io num_labels = 21 mask = np.array(Image.open(io.BytesIO(return_img))) plt.imshow(mask, vmin=0, vmax=num_labels-1, cmap='jet') plt.show()  This image is the segmentation mask that was created and it represents the motorbike and everything else is the background. [ As you can see from the code there are 21 possible labels and 2 were used in the mask, one for the motorbike and another for the background. What I would like to figure out now is how to print which labels were actually used in this mask out of the 21 possible options? Please let me know if you need any further information and any help is much appreciated. 

Answer: Somewhere you should have a mapping from label integers to label classes, e.g. label_map = {0: 'background', 1: 'motorbike', 2: 'train', ...}  If you are using the Pascal VOC dataset, that would be (1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle, 6=bus, 7=car , 8=cat, 9=chair, 10=cow, 11=diningtable, 12=dog, 13=horse, 14=motorbike, 15=person, 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor) - see here: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/segexamples/index.html Then you can simply use that map: used_classes = np.unique(mask) for cls in used_classes: print("Found class: {}".format(label_map[cls]))
Question: I have connected to Amazon S3 and am trying to retrieve data from the JSON content from multiple buckets using the below code. But I have to read only specific JSON files, but not all. How do I do it? Code: for i in bucket: try: result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/') content_object = s3.Object(i, "PROCESSED_BY/FILE_JSON/?Account.json") file_content = content_object.get()['Body'].read().decode('utf-8') json_content = json.loads(file_content) except KeyError: pass  Bucket structure example. test-eob/PROCESSED_BY/FILE_JSON/222-Account.json test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json test-eob/PROCESSED_BY/FILE_JSON/122-multi.json test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json  From the above list, I want to only read *-Account.json files. How can I achieve this?

Answer: There are several ways to do this in Python. For example, checking if 'stringA' is in 'stringB': list1=['test-eob/PROCESSED_BY/FILE_JSON/222-Account.json', 'test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json', 'test-eob/PROCESSED_BY/FILE_JSON/122-multi.json', 'test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json', 'test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json',] for i in list1: if 'Account' in i: print (i) else: pass
Question: poke_list = [... 'Charizard', '(Mega', 'Charizard', 'X)', '78', '130', ...] #1000+ values  Is it possible to merge strings that start with '(' and end with ')' and then reinsert it into the same list or a new list?  My desired output poke_list = [... 'Charizard (Mega Charizard X)', '78', '130', ...]

Answer: Another way to do it, slightly shorter than other solution poke_list = ['Bulbasaur', 'Charizard', '(Mega', 'Charizard', 'X)', '78', 'Pikachu', '(Raichu)', '130'] fixed = [] acc = fixed for x in poke_list: if x[0] == '(': acc = [fixed.pop()] acc.append(x) if x[-1] == ')': fixed.append(' '.join(acc)) acc = fixed if not acc is fixed: fixed.append(' '.join(acc)) print(fixed)  Also notice that this solution assumes that the broken list doesn't start with a parenthesis to fix, and also manage the case where an item has both opening and closing parenthesis (case excluded in other solution) The idea is to either append values to main list (fixed) or to some inner list which will be joined later if we have detected opening parenthesis. If the inner list was never closed when exiting the loop (likely illegal) we append it anyway to the fixed list when exiting the loop. This way of doing things if very similar to the transformation of a flat expression containing parenthesis to a hierarchy of lists. The code would of course be slightly different and should manage more than one level of inner list.
Question: I would like to create a custom preprocessing layer using the tf.keras.layers.experimental.preprocessing.PreprocessingLayer layer. In this custom layer, placed after the input layer, I would like to normalize my image using tf.cast(img, tf.float32) / 255. I tried to find some code or example showing how to create this preprocessing layer, but I couldn't find. Please, can someone provide a full example creating and using the PreprocessingLayer layer ?

Answer: If you want to have a custom preprocessing layer, actually you don't need to use PreprocessingLayer. You can simply subclass Layer Take the simplest preprocessing layer Rescaling as an example, it is under the tf.keras.layers.experimental.preprocessing.Rescaling namespace. However, if you check the actual implementation, it is just subclass Layer class Source Code Link Here but has @keras_export('keras.layers.experimental.preprocessing.Rescaling') @keras_export('keras.layers.experimental.preprocessing.Rescaling') class Rescaling(Layer): """Multiply inputs by `scale` and adds `offset`. For instance: 1. To rescale an input in the `[0, 255]` range to be in the `[0, 1]` range, you would pass `scale=1./255`. 2. To rescale an input in the `[0, 255]` range to be in the `[-1, 1]` range, you would pass `scale=1./127.5, offset=-1`. The rescaling is applied both during training and inference. Input shape: Arbitrary. Output shape: Same as input. Arguments: scale: Float, the scale to apply to the inputs. offset: Float, the offset to apply to the inputs. name: A string, the name of the layer. """ def __init__(self, scale, offset=0., name=None, **kwargs): self.scale = scale self.offset = offset super(Rescaling, self).__init__(name=name, **kwargs) def call(self, inputs): dtype = self._compute_dtype scale = math_ops.cast(self.scale, dtype) offset = math_ops.cast(self.offset, dtype) return math_ops.cast(inputs, dtype) * scale + offset def compute_output_shape(self, input_shape): return input_shape def get_config(self): config = { 'scale': self.scale, 'offset': self.offset, } base_config = super(Rescaling, self).get_config() return dict(list(base_config.items()) + list(config.items()))  So it proves that Rescaling preprocessing is just another normal layer. The main part is the def call(self, inputs) function. You can create whatever complicated logic to preprocess your inputs and then return. A easier documentation about custom layer can be find here In a nutshell, you can do the preprocessing by layer, either by Lambda which for simple operation or by subclassing Layer to achieve your goal.
Question: I have python3 3.6.9 on Kubuntu 18.04. I have installed fastapi using pip3 install fastapi. I'm trying to test drive the framework through its official documentation and I'm in the relational database section of its guide. In schemas.py: from typing import List from pydantic import BaseModel class VerseBase(BaseModel): AyahText: str NormalText: str class Verse(VerseBase): id: int class Config: orm_mode = True  VS code highlights an error in from pydantic import BaseModel and it tells that: No name 'BaseModel' in module 'pydantic'. Additionally, when I try to run uvicorn main:app reload I have gotten the following error:  File "./main.py", line 6, in <module> from . import crud, models, schemas ImportError: attempted relative import with no known parent package   I have tried to renstall pydantic using pip3 but it tells me that: Requirement already satisfied: dataclasses>=0.6; python_version < "3.7" in ./.local/lib/python3.6/site-packages (from pydantic) (0.7) 

Answer: The problem of highlighting in VS code may be a problem due to the fact that you did not open the folder. It's quite annoying as it happens often to me as well (and I have basically your same config). Regarding the second problem you mention, it is probably due to the fact that the folder in which the script lays, does not have a __init__.py file. If you add it, it should work since python will interpret the folder as a module. As an alternative, you could try to import with the full path from the top folder (e.g. from app.module.main import app). For more information about modules see the following links: Python 3.8 Modules Real Python
Question: I'm trying to get the files from specific folders in s3 Buckets: I have 4 buckets in s3 with the following names: 1 - 'PDF' 2 - 'TXT' 3 - 'PNG' 4 - 'JPG'  The folder structure for all s3 buckets looks like this: 1- PDF/analysis/pdf-to-img/processed/files 2- TXT/report/processed/files 3- PNG/analysis/reports/png-to-txt/processed/files 4- JPG/jpg-to-txt/empty  I have to check if this folder prefix processed/files is present in the bucket, and if it is present, I'll read the files present in those directories, else I'll ignore them.  Code: buckets = ['PDF','TXT','PNG','JPG'] client = boto3.client('s3') for i in bucket: result = client.list_objects(Bucket=i,Prefix = 'processed/files', Delimiter='/') print(result)  I can enter into each directory if the folder structure is same, but how can I handle this when the folder structure varies for each bucket?

Answer: This is maybe a lengthy process.  buckets = ['PDF','TXT','PNG','JPG'] s3_client = getclient('s3') for i in buckets: result = s3_client.list_objects(Bucket= i, Prefix='', Delimiter ='') contents = result.get('Contents') for content in contents: if 'processed/files/' in content.get('Key'): print("Do the process")  You can get the list of directories from the s3 bucket. If it contains the required folder do the required process.
Question:  I am using python 3 on windows 7, selenium, chromedriver version 84 (latest) to automate my chrome browser.  I am using this script: from selenium import webdriver #import chromedriver_binary # Adds chromedriver binary to path driver = webdriver.Chrome() driver.get("http://www.python.org")  and I always get this error upon running it. Traceback (most recent call last): File "D:\Huzefa\Desktop\zzzzzz.py", line 4, in <module> driver = webdriver.Chrome() File "C:\Users\Huzefa\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 81, in __init__ desired_capabilities=desired_capabilities) File "C:\Users\Huzefa\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__ self.start_session(capabilities, browser_profile) File "C:\Users\Huzefa\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "C:\Users\Huzefa\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\Huzefa\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 84  My ChromeDriver is in path. Also i have used other versions of chromedriver but i am not able to navigate to a website!

Answer: Your ChromeDriver version and your installed version of Chrome need to match up. You are using ChromeDriver for Chrome version 84, which at the time of this answer, is a beta (non-stable) build of Chrome; you're probably not using it. Likely you're on version 83. Check your Chrome version (Help -> About) and then find the correct ChromeDriver release. You could instead use webdriver-manager which can handle this for you.
Question: I'm assuming I'm doing something wrong here, but I'm working on a project in Pycharm, which notified me when using the ndarray.max() function that initial was undefined (parameter 'initial' unfilled). Looking at the documentation, it does show that there is no default value for initial argument. When ctrl-clicking the ndarray.max() function in Pycharm, opens the following function:  def max(self, axis=None, out=None, keepdims=False, initial, *args, **kwargs): # real signature unknown; NOTE: unreliably restored from __doc__ """ a.max(axis=None, out=None, keepdims=False, initial=<no value>, where=True) Return the maximum along a given axis. Refer to `numpy.amax` for full documentation. See Also -------- numpy.amax : equivalent function """ pass  Which appears to not even do anything. Either way, the code works, only an IDE error is given. Am I using the wrong function? I know there's amax and max, as well as the package level numpy.max, but the above seems to be unwanted behaviour. If this is a bug, I wouldn't know how to report it / start an issue or whatever, haha.

Answer: it appears empty because it's not implemented in python, probably C/C++, as you can figure out from # real signature unknown; NOTE: unreliably restored from __doc__ - it's just a hint for you what parameter this function has. It's not even valid python ;)  Basing on documentation of amax:  initial scalar, optional The minimum value of an output element. Must be present to allow computation on empty slice. See reduce for details.  You'd better pass something to initial
Question: I have a website that shows the prices of items in a video game. Currently, I have an "auto-refresh" script that refreshes the page every 5 seconds, but it is a bit annoying as every time you search for a product, it removes your search because the page refreshes. I would like to update the numbers in my table without refreshing the page for the user. I read something about 'updating the DOM', in javascript but didn't get it. Here is the link to my website: http://xeltool.com/ And here is my python code: @app.route('/bprices', methods=['GET']) def bPrices(): f = requests.get( 'https://api.hypixel.net/skyblock/bazaar?key=[cannot show]').json() products = [ { "id": product["product_id"], "sell_price": product["sell_summary"][:1], "buy_price": product["buy_summary"][:1], "sell_volume": product["quick_status"]["sellVolume"], "buy_volume": product["quick_status"]["buyVolume"], } for product in f["products"].values() ] return render_template("bprices.html", products=products)  And here is my HTML code: <div class="container"> <div class="search_div"> <input type="text" onkeyup="myFunction()" id="myInput" title="Type in a product" class="search-box" placeholder="Search for a product..." /> <button class="search-btn"><i class="fas fa-search"></i></button> </div> <table id="myTable" class="table table-striped table-bordered table-sm table-dark sortable" cellspacing="0" > <thead> <tr> <th aria-label="Product Name" data-balloon-pos="up">Product</th> <th aria-label="Product's buy price" data-balloon-pos="up"> Buy Price </th> <th aria-label="Product's sell price" data-balloon-pos="up"> Sell Price </th> <th aria-label="Product's buy volume" data-balloon-pos="up"> Buy Volume </th> <th aria-label="Product's sell volume" data-balloon-pos="up"> Sell Volume </th> <th> Margin </th> </tr> </thead> <tbody> {% for product in products %} <tr> <td>{{ product.id|replace("_", ' ')|lower()|title() }}</td> {% for buy in product.buy_price %} <td>{{ buy.pricePerUnit }}</td> {% for sell in product.sell_price %} <td>{{ sell.pricePerUnit }}</td> <td>{{ product.buy_volume| numberFormat }}</td> <td>{{ product.sell_volume| numberFormat }}</td> {% set margin = buy.pricePerUnit - sell.pricePerUnit%} {% set marginPer = margin/buy.pricePerUnit * 100%} <td aria-label="{{ marginPer|round(1, 'floor') }} % " data-balloon-pos="right" > {{ margin|round(1, 'floor')}} </td> {% endfor %}{% endfor %} </tr> {% endfor %} </tbody> </table> </div>  If you NEED the API to test this out, I can provide a link to it :)

Answer: You have 3 options:  AJAX - https://www.w3schools.com/js/js_ajax_intro.asp SSE - https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events Websocket - https://developer.mozilla.org/en-US/docs/Glossary/WebSockets  I think the best option in your case is SSE since the server knows that the price was changed so it can push it to the clients.
Question: I am using Pandas to read CSV file data, but the CSV module is also there to manage the CSV file. What is the difference between these both? What are the cons of using Pandas over the CSV module?

Answer: Based upon benchmarks  CSV is faster to load data for smaller datasets (< 1K rows) Pandas is several times faster for larger datasets  Code to Generate Benchmarks Benchmarks
Question: Given a directory tests with a few subdirectories each containing test modules, how can one create a pytest fixture to be run before each test found in a particular subdirectory only? tests ├── __init__.py ├── subdirXX │ ├── test_module1.py │ ├── test_module2.py │ ├── __init__.py ├── subdirYY │ ├── test_module3.py │ ├── test_module4.py │ ├── __init__.py  I'd like to have a fixture that will run before each test found in modules within the subdirYY only (in this case, in modules test_module3.py and test_module4.py). I currently have a fixture defined twice, once inside each module within the subdirYY which works but is redundant: @pytest.fixture(autouse=True) def clean_directory(): ...  If this is not possible to achieve, each of the tests within the subdirYY is decorated with a custom mark (@pytest.mark.mycustommark) so making sure that a certain fixture will run before each test marked with a particular custom mark is a viable option, too.

Answer: Put your autouse fixture in a conftest.py file inside subdirYY. For more information, see the pytest docs about sharing fixtures and the docs on autouse fixtures which specifically mention conftest.py:  if an autouse fixture is defined in a conftest.py file then all tests in all test modules belows its directory will invoke the fixture.
Question: When trying to run my Discord bot I get this error: raise ClientConnectorCertificateError( aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host discordapp.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)')]  It just started happening out of nowhere and happens every time. I'm using Python 3.8 on Windows 10. What does this mean and how do I fix it?

Answer: To fix this:  Go to discord.com with Internet Explorer (Ran as Administrator) Click the lock on the top right Click view certificates Install one  PS: If your antivirus is active for the web browser and this solution doesn't work, try disabling it and try again.
Question: When converting code from Python 2 to Python 3 one issue is that the behaviour when testing strings and bytes for equality has changed. For example: foo = b'foo' if foo == 'foo': print("They match!")  prints nothing on Python 3 and "They match!" on Python 2. In this case it is easy to spot but in many cases the check is performed on variables which may have been defined elsewhere so there is no obvious type information. I would like to make the Python 3 interpreter give an error whenever there is an equality test between string and bytes rather than silently conclude that they are different. Is there any way to accomplish this?

Answer: There is an option, -b, you can pass to the Python interpreter to cause it to emit a warning or error when comparing byte / str. > python --help usage: /bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Options and arguments (and corresponding environment variables): -b : issue warnings about str(bytes_instance), str(bytearray_instance) and comparing bytes/bytearray with str. (-bb: issue errors)  This produces a BytesWarning as seen here: > python -bb -i Python 3.8.0 Type "help", "copyright", "credits" or "license" for more information. >>> v1 = b'foo' >>> v2 = 'foo' >>> v1 == v2 Traceback (most recent call last): File "<stdin>", line 1, in <module> BytesWarning: Comparison between bytes and string
Question: I am trying to check if a number is in range of integers and returns a number based on which range it lies. I was wondering if is there a better and more efficient way of doing this: def checkRange(number): if number in range(0, 5499): return 5000 elif number in range(5500, 9499): return 10000 elif number in range(9500, 14499): return 15000 elif number in range(14500, 19499): return 20000 elif number in range(19500, 24499): return 25000 elif number in range(24500, 29499): return 30000 elif number in range(29500, 34499): return 35000 elif number in range(34500, 39499): return 40000 elif number in range(39500, 44499): return 45000  This felt like a waste of resources and would greatly appreciate if there is a better way to do this.

Answer: Since you have continuous, sorted ranges, a quicker and less verbose way to do this, is to use the bisect module to find the index in a list of breakpoints and then use it to get the corresponding value from a list of values: import bisect break_points = [5499, 9499, 14499, 19499, 24499, 29499, 34499, 39499, 44499] values = [5000, 10000, 15000, 20000, 25000, 30000, 35000, 40000, 45000] n = 10000 index = bisect.bisect_left(break_points, n) values[index] # 15000  You'll need to test for n values that exceed the last breakpoint if that's a possibility. Alternatively you can add a default value to the end of the values list.
Question: Using djangorestframework_simplejwt library, when POST to a custom view #urls.py path('api/token/', MyTokenObtainPairView.as_view(), name='token_obtain'), #views.py class MyTokenObtainPairView(TokenObtainPairView): serializer_class = MyTokenObtainPairSerializer  I'm able to get a the following access token eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNTkwOTEwNjg0LCJqdGkiOiI3M2MxYmZkOWNmMGY0ZjI3OTY4MGY0ZjhlYjA1NDQ5NyIsInVzZXJfaWQiOjExfQ.5vs0LmNGseU6rtq3vuQyApupxhQM3FBAoKAq8MUukIBOOYfDAV9guuCVEYDoGgK6rdPSIq2mvcSxkILG8OH5LQ  By going to https://jwt.io/ I can see the payload is currently { "token_type": "access", "exp": 1590910684, "jti": "73c1bfd9cf0f4f279680f4f8eb054497", "user_id": 11 }   So, we can see that the second part of the token is the payload - containing the claims. I've explored how to add more information to the Response body and now would like to know how to customize the Payload data by adding iat claim, username and today's date.

Answer: As you already created a subclass for the desired view (MyTokenObtainPairView) and a subclass for its corresponding serializer (MyTokenObtainPairSerializer), add the following to the serializer class MyTokenObtainPairSerializer(TokenObtainPairSerializer): ... @classmethod def get_token(cls, user): token = super().get_token(user) # Add custom claims token['iat'] = datetime.datetime.now() token['user'] = user.username token['date'] = str(datetime.date.today()) return token  Then, when you POST to that same location, you'll get an access token like this eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNTkwOTE0MTk4LCJqdGkiOiJhZDZmNzZhZjFmOGU0ZWJlOGI2Y2Y5YjQ4MGQzZjY2MiIsInVzZXJfaWQiOjExLCJpYXQiOjE1OTA5MTc0OTgsInVzZXIiOiJ0aWFnbyIsImRhdGUiOiIyMDIwLTA1LTMxIn0.-5U9P-WWmhlOenzCvc6b7_71Tz17LyNxe_DOMwwqH4RqrNsilVukEcZWFRGupLHRZjIvPya2QJGpiju9ujzQuw  Using JWT you can see the Payload changing accordingly  { "token_type": "access", "exp": 1590914198, "jti": "ad6f76af1f8e4ebe8b6cf9b480d3f662", "user_id": 11, "iat": 1590917498, "user": "tiago", "date": "2020-05-31" }