eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have 2 modules in my project: first works with image in bytes format, second requires skimage object. I need to combine them. I have this code: import io from PIL import Image import skimage.io area = (...) image = Image.open(io.BytesIO(image_bytes)) image = Image.crop(area) image = skimage.io.imread(image)  But i get this error:  How can i convert an image (object/variable) to skimage? I don't necessarily need PIL Image, this is just one way to work with bytes image, cause i need to crop my image Thanks!

Answer: Scikit-image works with images stored as Numpy arrays - same as OpenCV and wand. So, if you have a PIL Image, you can make a Numpy array for scikit-image like this: # Make Numpy array for scikit-image from "PIL Image" na = np.array(YourPILImage)  Just in case you want to go the other way, and make a PIL Image from a Numpy array, you can do: # Make "PIL Image" from Numpy array pi = Image.fromarray(na)
Question: We are using Django as backend for a website that provides various things, among others using a Neural Network using Tensorflow to answer to certain requests. For that, we created an AppConfig and added loading of this app config to the INSTALLED_APPS in Django's settings.py. This AppConfig then loads the Neural Network as soon as it is initialized: settings.py: INSTALLED_APPS = [ ... 'bert_app.apps.BertAppConfig', ]  .../bert_apps/app.py: class BertAppConfig(AppConfig): name = 'bert_app' if 'bert_app.apps.BertAppConfig' in settings.INSTALLED_APPS: predictor = BertPredictor() #loads the ANN.  Now while that works and does what it should, the ANN is now loaded for every single command run through manage.py. While we of course want it to be executed if you call manage.py runserver, we don't want it to be run for manage.py migrate, or manage.py help and all other commands. I am generally not sure if this is the proper way how to load an ANN for a Django-Backend in general, so does anybody have any tips how to do this properly? I can imagine that loading the model on startup is not quite best practice, and I am very open to suggestions on how to do that properly instead. However, there is also some other code besides the actual model-loading that also takes a few seconds and that is definitely supposed to be executed as soon as the server starts up (so on manage.py runserver), but also not on manage.py help (as it takes a few seconds as well), so is there some quick fix for how to tell Django to execute it only on runserver and not for its other commands?

Answer: I had a similar problem, solved it with checking argv. class SomeAppConfig(AppConfig): def ready(self, *args, **kwargs): is_manage_py = any(arg.casefold().endswith("manage.py") for arg in sys.argv) is_runserver = any(arg.casefold() == "runserver" for arg in sys.argv) if (is_manage_py and is_runserver) or (not is_manage_py): init_your_thing_here()  Now a bit closer to the if not is_manage_py part: in production you run your web server with uwsgi/uvicorn/..., which is still a web server, except it's not run with manage.py. Most likely, it's the only thing that you will ever run without manage.py  Use AppConfig.ready() - it's intended for it:  Subclasses can override this method to perform initialization tasks such as registering signals. It is called as soon as the registry is fully populated. - [django documentation]  To get your AppConfig back, use: from django.apps import apps apps.get_app_config(app_name) # apps.get_app_configs() # all
Question: Suppose you have a simple class like A below. The comparison methods are all virtually the same except for the comparison itself. Is there a shortcut around declaring the six methods in one method such that all comparisons are supported, something like B? I ask mainly because B seems more Pythonic to me and I am surprised that my searching didn't find such a route. class A: def __init__(self, number: float, metadata: str): self.number = number self.metadata = metadata def __lt__(self, other): return self.number < other.number def __le__(self, other): return self.number <= other.number def __gt__(self, other): return self.number > other.number def __ge__(self, other): return self.number >= other.number def __eq__(self, other): return self.number == other.number def __ne__(self, other): return self.number != other.number class B: def __init__(self, number: float, metadata: str): self.number = number self.metadata = metadata def __compare__(self, other, comparison): return self.number.__compare__(other.number, comparison) 

Answer: The functools module provides the total_ordering decorator which is meant to provide all comparison methods, given that you provide at least one from __lt__(), __le__(), __gt__(), or __ge__(). See this answer from Martijn Pieters.
Question: I have a dockerfile, where my image is python:3.7-alpine. In my project, I use a git repository I need to download. Is there any way to do that ? My Dockerfile : FROM python:3.7-alpine ENV DOCKER_APP True COPY requirements.txt . RUN pip install -r requirements.txt COPY . app/ WORKDIR app/ ENTRYPOINT ["python3", "main.py"]  my requirements : certifi==2020.6.20 requests==2.24.0 urllib3==1.25.10 git+https://github.com/XXX/YYY  Thank

Answer: add the following to your dockerfile RUN apk update RUN apk add git
Question: Hi I'm new to pyspark and I'm trying to convert pyspark.sql.dataframe into list of dictionaries. Below is my dataframe, the type is <class 'pyspark.sql.dataframe.DataFrame'>: +------------------+----------+------------------------+ | title|imdb_score|Worldwide_Gross(dollars)| +------------------+----------+------------------------+ | The Eight Hundred| 7.2| 460699653| | Bad Boys for Life| 6.6| 426505244| | Tenet| 7.8| 334000000| |Sonic the Hedgehog| 6.5| 308439401| | Dolittle| 5.6| 245229088| +------------------+----------+------------------------+  I would like to convert it into: [{"title":"The Eight Hundred", "imdb_score":7.2, "Worldwide_Gross(dollars)":460699653}, {"title":"Bad Boys for Life", "imdb_score":6.6, "Worldwide_Gross(dollars)":426505244}, {"title":"Tenet", "imdb_score":7.8, "Worldwide_Gross(dollars)":334000000}, {"title":"Sonic the Hedgehog", "imdb_score":6.5, "Worldwide_Gross(dollars)":308439401}, {"title":"Dolittle", "imdb_score":5.6, "Worldwide_Gross(dollars)":245229088}]  How should I do this? Thanks in advance!

Answer: You can map each row into a dictionary and collect the results: df.rdd.map(lambda row: row.asDict()).collect()
Question: I have created a dictionary in my Python application where I save the data and I have two tasks that run concurrently and get data from external APIs. Once they get the data, they update the dictionary - each with a different key in the dictionary. I want to understand if the dictionary is async safe or do I need to put a lock when the dictionary is read/updated? The tasks also read the last saved value each time. my_data = {} asyncio.create_task(call_func_one_coroutine) asyncio.create_task(call_func_two_coroutine) async def call_func_one_coroutine(): data = await goto_api_get_data() my_data['one'] = data + my_data['one'] async def call_func_two_coroutine(): data = await goto_api_another_get_data() my_data['two'] = data + my_data['two'] 

Answer:  I want to understand if the dictionary is async safe or do I need to put a lock when the dictionary is read/updated?  Asyncio is based on cooperative multitasking, and can only switch tasks at an explicit await expression or at the async with and async for statements. Since update of a single dictionary can never involve awaiting (await must completes before the update begins), it is effectively atomic as far as async multitasking is concerned, and you don't need to lock it. This applies to all data structures accessed from async code. To take another example where there is no problem: # correct - there are no awaits between two accesses to the dict d key = await key_queue.get() if key in d: d[key] = calc_value(key)  An example where a dict modification would not be async-safe would involve multiple accesses to the dict separated by awaits. For example: # incorrect, d[key] could appear while we're reading the value, # in which case we'd clobber the existing key if key not in d: d[key] = await read_value()  To correct it, you can either add another check after the await, or use an explicit lock: # correct (1), using double check if key not in d: value = await read_value() # Check again whether the key is vacant. Since there are no awaits # between this check and the update, the operation is atomic. if key not in d: d[key] = value # correct (2), using a shared asyncio.Lock: async with d_lock: # Single check is sufficient because the lock ensures that # no one can modify the dict while we're reading the value. if key not in d: d[key] = await read_value()
Question: BACKGROUND I am trying to download GFS weather data netcdf4 files via xarray & OPeNDAP. Big thanks to Vorticity0123 for their prior post, which allowed me to get the bones of the python script sorted (as below). PROBLEM Thing is, the GFS dataset has 195 data variables, But I don't require the majority, I only need ten of them.  ugrd100m, vgrd100m, dswrfsfc, tcdcclm, tcdcblcll, tcdclcll, tcdcmcll, tcdchcll, tmp2m, gustsfc  HELP REQUESTED I've gone through the xarray readthedocs page and elsewhere, but I couldn't figure out a way to narrow down my dataset to only the ten data variables. Does anyone know how to narrow down the list of variables in a dataset? PYTHON SCRIPT import numpy as np import xarray as xr # File Details dt = '20201124' res = 25 step = '1hr' run = '{:02}'.format(18) # URL URL = f'http://nomads.ncep.noaa.gov:80/dods/gfs_0p{res}_{step}/gfs{dt}/gfs_0p{res}_{step}_{run}z' # Load data dataset = xr.open_dataset(URL) time = dataset.variables['time'] lat = dataset.variables['lat'][:] lon = dataset.variables['lon'][:] lev = dataset.variables['lev'][:] # Narrow Down Selection time_toplot = time lat_toplot = np.arange(-43, -17, 0.5) lon_toplot = np.arange(135, 152, 0.5) lev_toplot = np.array([1000]) # Select required data via xarray dataset = dataset.sel(time=time_toplot, lon=lon_toplot, lat=lat_toplot) print(dataset) 

Answer: You can use the dict-like syntax of xarray. variables = [ 'ugrd100m', 'vgrd100m', 'dswrfsfc', 'tcdcclm', 'tcdcblcll', 'tcdclcll', 'tcdcmcll', 'tcdchcll', 'tmp2m', 'gustsfc' ] dataset[variables]  Gives you: <xarray.Dataset> Dimensions: (lat: 721, lon: 1440, time: 121) Coordinates: * time (time) datetime64[ns] 2020-11-24T18:00:00 ... 2020-11-29T18:00:00 * lat (lat) float64 -90.0 -89.75 -89.5 -89.25 ... 89.25 89.5 89.75 90.0 * lon (lon) float64 0.0 0.25 0.5 0.75 1.0 ... 359.0 359.2 359.5 359.8 Data variables: ugrd100m (time, lat, lon) float32 ... vgrd100m (time, lat, lon) float32 ... dswrfsfc (time, lat, lon) float32 ... tcdcclm (time, lat, lon) float32 ... tcdcblcll (time, lat, lon) float32 ... tcdclcll (time, lat, lon) float32 ... tcdcmcll (time, lat, lon) float32 ... tcdchcll (time, lat, lon) float32 ... tmp2m (time, lat, lon) float32 ... gustsfc (time, lat, lon) float32 ... Attributes: title: GFS 0.25 deg starting from 18Z24nov2020, downloaded Nov 24 ... Conventions: COARDS\nGrADS dataType: Grid history: Sat Nov 28 05:52:44 GMT 2020 : imported by GrADS Data Serve...
Question: I'm trying to get number of days between two dates using below function df['date'] = pd.to_datetime(df.date) # Creating a function that returns the number of days def calculate_days(date): today = pd.Timestamp('today') return today - date # Apply the function to the column date df['days'] = df['date'].apply(lambda x: calculate_days(x))  The results looks like this  153 days 10:16:46.294037  but I want it to say 153. How do I handle this?

Answer: For performance you can subtract values without apply for avoid loops use Series.rsub for subtract from rigth side: df['date'] = pd.to_datetime(df.date) df['days'] = df['date'].rsub(pd.Timestamp('today')).dt.days  What working like: df['days'] = (pd.Timestamp('today') - df['date']).dt.days  If want use your solution: df['date'] = pd.to_datetime(df.date) def calculate_days(date): today = pd.Timestamp('today') return (today - date).days df['days'] = df['date'].apply(lambda x: calculate_days(x))  Or: df['date'] = pd.to_datetime(df.date) def calculate_days(date): today = pd.Timestamp('today') return (today - date) df['days'] = df['date'].apply(lambda x: calculate_days(x)).dt.days
Question: When using pythons random.shuffle function, I noticed it went significantly faster to use sorted(l, key=lambda _: random.random()) than random.shuffle(l). As far as I understand, both ways produce completely random lists, so why does shuffle take so much longer? Below are the times using timeit module. from timeit import timeit setup = 'import random\nl = list(range(1000))' # 5.542 seconds print(timeit('random.shuffle(l)', setup=setup, number=10000)) # 1.878 seconds print(timeit('sorted(l, key=lambda _: random.random())', setup=setup, number=10000)) 

Answer: On CPython (the reference interpreter) random.shuffle is implemented in Python (and implemented in terms of _randbelow, itself a Python wrapper around getrandbits, the C level function that ultimately implements it, and which can end up being called nearly twice as often as strictly necessary in an effort to ensure the outputs are unbiased); sorted (and random.random) are implemented in C. The overhead of performing work in Python is higher than performing similar work in C.
Question: I have a dataframe with a column whose values look something like:  YEAR_TORONTO 0 <NA> 1 2016 2 <NA> 3 1999  I need to check each element of this dataframe individually via a for loop for other reasons outside this segment of code, so I'm looking for solutions that comply with my implementation. Essentially the code I have at the moment to check the presence of is: if np.isnan(df.get("YEAR_TORONTO")[row]):  This leads to me getting the following error for the <NA> values: boolean value of NA is ambiguous  Any idea of what I can do to fix this error? Help much appreciated

Answer: As sammywemmy said, pd.isna() should work. >>> d = pd.Series([1,2,pd.NA,3]) >>> d 0 1 1 2 2 <NA> 3 3 dtype: object >>> d.isna() 0 False 1 False 2 True 3 False dtype: bool
Question: Can someone say if "²" is a symbol or a digit? (alt+1277, power of two) print("²".isdigit()) # True print("²".isnumeric()) # True  Because Python says it's a digit, but it's not actually a digit. Am I wrong? Or it's a bug?

Answer: It is explicitly documented as a digit:  str.isdigit() Return True if all characters in the string are digits and there is at least one character, False` otherwise. Digits include decimal characters and digits that need special handling, such as the compatibility superscript digits. This covers digits which cannot be used to form numbers in base 10, like the Kharosthi numbers. Formally, a digit is a character that has the property value Numeric_Type=Digit or Numeric_Type=Decimal.  Regarding Numeric_Type, this is defined by Unicode:  Numeric_Type=Digit Variants of positional decimal characters (Numeric_Type=Decimal) or sequences thereof. These include super/subscripts, enclosed, or decorated by the addition of characters such as parentheses, dots, or commas.
Question: I am using the Scibert pretrained model to get embeddings for various texts. The code is as follows: from transformers import * tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased', model_max_length=512, truncation=True) model = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased')  I have added both the max length and truncation parameters to tokenizers, but unfortunately, they don't truncate the results.If I run a longer text through the tokenizer: inputs = tokenizer("""long text""")  I get the following error:  Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors  Now obviously I can't run this through the model due to having too long sequences of tensors. What is the easiest way to truncate the input to fit the maximum sequence length of 512?

Answer: truncation is not a parameter of the class constructor (class reference), but a parameter of the __call__ method. Therefore you should use: tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased', model_max_length=512) len(tokenizer(text, truncation=True).input_ids)  Output: 512
Question: Is there a way to increase the stdout buffer size from 8182 in Python or to delay the flush until I actually call flush? Things I've tried that don't work:  I can get around this issue on Windows because I can access the buffer directly (e.g. see my answer to this post). But this doesn't work for Unix. I can increase the buffer size for a file by passing buffer to the constructor, however stdout is already constructed. Turning off buffering (python -u) obviously makes things worse! Using a temporary buffer encounters the same problems - stdout is flushed after every 8192nd byte is copied from the temporary buffer.  Rationale: The aim here is to reduce console flickering. Buffering everything, as per this question indeed works, for instance when I try in C or by using the Windows API in Python, but the 8182 limit in Python seems to be causing problems that I can't get around on Unix.

Answer: Found the answer, actually very simple: my_stdout = open( 1, "w", buffering = 100000 )   1 is the fileno for stdout. sys.stdout = my_stdout can be used to make the change to the default print target. I've only tested this on Unix.
Question: When I install my example module in the local environment, python is able to find it when the module is imported. Whereas, when executed by Github Actions, the workflow fails and the reported error is that my module (ci-test) is not installed. main.yaml:  - name: Install ci-test package run: | python setup.py build python setup.py install python -c "import ci_test"  The full yaml file is located here. And the error output of Github Actions is: Installed /home/runner/.local/lib/python3.7/site-packages/ci_test-0.0.1-py3.7.egg Processing dependencies for ci-test==0.0.1 Finished processing dependencies for ci-test==0.0.1 Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'ci_test' Error: Process completed with exit code 1. 

Answer: The issue is not related to github action. When looking to your repository, the repository is organized this way. ci-test/ |-- requirements.txt |-- setup.py |-- src/ | |-- ci_test/ | | |-- app.py | | |-- __init__.py | | |-- main.py |-- tests/ | |-- app_test.py | |-- __init__.py | |-- main_test.py  And in your setup.py you describe your packages as: packages=['src/ci_test', 'tests']  Your ci_test package can be import with the following path: import src.ci_test This question have some best practices about python project structure: What is the best project structure for a Python application?
Question: When coding I really like to use list comprehensions to transform data and I try to avoid for loops. Now I discovered that the walrus operator can be really handy for this, but when I try to use it in my code it doesn't seem to work. I've got the following code and want to transform the strings containing data about the timestamps into datetime objects in one easy line, but I get an syntax error and I am not sure what the right syntax would be, does anyone know what I did wrong? from datetime import datetime timestamps = ['30:02:17:36', '26:07:44:25','25:19:30:38','25:07:40:47','24:18:29:05','24:06:13:15','23:17:36:39', '23:00:14:52','22:07:04:33','21:15:42:20','21:04:27:53', '20:12:09:22','19:21:46:25'] timestamps_dt = [ datetime(days=day,hours=hour,minutes=mins,seconds=sec) for i in timestamps day,hour,mins,sec := i.split(':') ] 

Answer: Since Walrus operator does not support values unpacking, the operation day,hour,mins,sec := i.split(':') is invalid. Walrus operator is recommended to be used mostly in logic comparison, especially when you need to reuse a variable in comparison. Therefore, I would argue that for this case, a simple datetime.strptime() would be better for this case. If you must use walrus comparison in your list comprehension, you may do from datetime import datetime timestamps = ['30:02:17:36', '26:07:44:25','25:19:30:38','25:07:40:47','24:18:29:05','24:06:13:15','23:17:36:39', '23:00:14:52','22:07:04:33','21:15:42:20','21:04:27:53', '20:12:09:22','19:21:46:25'] timestamps_dt = [ datetime(2020,11, *map(int, time)) # map them from str to int for i in timestamps if (time := i.split(':')) # assign the list to the variable time ] print(timestamps_dt)  But then it will lead to a question why not just, timestamps_dt = [ datetime(2020,11, *map(int, i.split(':'))) for i in timestamps ]  Reference PEP-572
Question: I am taking a programming class in college and one of the exercises in the problem sheet was to write this code: number = int(input()) x = 0 y = 0 for n in range(number): if n % 2 == 0: x += n else: y += n print(x) print(y)  using only one "for" loop, and no "while" or "if".  The purpose of the code is to find the sum of the even and the sum of the odd numbers from zero to the number inputted and print it to the screen.  Be reminded that at this time we aren't supposed to know about functions. I've been trying for a long time now and can't seem to find a way of doing it without using "if" statements to know if the loop variable is even or odd.

Answer: for n in range(number): x += (1 - n % 2) * n y += (n % 2) * n
Question: I have a Django Proj running in Docker Container My Debug=True but docker up logging doesn't show any print('xxxx') output. Is there a way to fix it? thanks!

Answer: After a long search I found this https://serverfault.com/a/940357 Add flush=True print(datetime.now(), flush=True)  Or add PYTHONUNBUFFERED: 1 to docker-compose.yml which is added by PyCharm by default version: '3.6' services: test: .... environment: PYTHONUNBUFFERED: 1 # <--- ....
Question: I have the following data frame: df = pd.DataFrame({'color': ['Yellow', 'Green', 'Red', 'Orange'], 'weight': [0.5, 4, 1, 10]}, index = ['Banana','Melon','Apple','Pumpkin']) which looks like this: Banana Yellow 0.5 Melon Green 4.0 Apple Red 1.0 Pumpkin Orange 10.0  What I'm trying to do is access the integer locations of a subset of 'Banana','Melon','Apple','Pumpkin'. I know that pandas.index.get_loc is a way of doing this for a single element of index. For example: df.index.get_loc('Apple') returns 2. However, I would like to do this for multiple elements at once. I tried, among other things, df.index.get_loc('Apple','Pumpkin'), hoping to get the list [2,3], but this gave the following error: ValueError: Invalid fill method. Expecting pad (ffill), backfill (bfill) or nearest. Got pumpkin Is there a function that will allow me to get multiple integer locations at once?

Answer: Try with get_indexer which accept list like input df.index.get_indexer(['Apple','Pumpkin']) Out[104]: array([2, 3], dtype=int32)
Question: I have a pandas data frame that consists of 4 rows, the English rows contain news titles, some rows contain non-English words like this one **SheÃ¢â‚¬â„¢s the Hollywood Power Behind Those ...**  I want to remove all rows like this one, so all rows that contain at least non-English characters in the Pandas data frame.

Answer: If using Python >= 3.7: df[df['col'].map(lambda x: x.isascii())]  where col is your target column.  Data: df = pd.DataFrame({ 'colA': ['**SheÃ¢â‚¬â„¢s the Hollywood Power Behind Those ...**', 'Hello, world!', 'Cainã', 'another value', 'test123*', 'âbc'] }) print(df.to_markdown())  | | colA | |---:|:------------------------------------------------------| | 0 | **SheÃ¢â‚¬â„¢s the Hollywood Power Behind Those ...** | | 1 | Hello, world! | | 2 | Cainã | | 3 | another value | | 4 | test123* | | 5 | âbc |  Identifying and filtering strings with non-English characters (see the ASCII printable characters): df[df.colA.map(lambda x: x.isascii())]  Output:  colA 1 Hello, world! 3 another value 4 test123*   Original approach was to use a user-defined function like this: def is_ascii(s): try: s.encode(encoding='utf-8').decode('ascii') except UnicodeDecodeError: return False else: return True