eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe similar to the one below, and I would like to create a new variable which contains true/false if for each project the sector "a" has been covered at least once. I'm trying with the group.by() function, and wanted to use the .transform() method but since my data is text, I don't know how to use it.  project sector 01 a 01 b 02 b 02 b 03 a 03 a project sector new_col 01 a true 01 b true 02 b false 02 b false 03 a true 03 a true 

Answer: You could try the following: df['new_col'] = df.groupby('project')['sector'].transform(lambda x: (x == 'a').any() )  This will group by project and check if any 'a' is in the groups sectors
Question: I have an ordered Python list of forms: [1, 2, 3, 4, 5, 12, 13, 14, 15, 20, 21, 22, 23, 30, 35, 36, 37, 38, 39, 40]  How can I group together consecutive numbers in a list. A group like this: [[1, 2, 3, 4, 5], [12, 13, 14, 15], [20, 21, 22, 23,], [30], [35, 36, 37, 38, 39, 40]]  I tried using groupby from here but was not able to tailor it to my need. Thanks,

Answer: You could use negative indexing: def group_by_missing(seq): if not seq: return seq grouped = [[seq[0]]] for x in seq[1:]: if x == grouped[-1][-1] + 1: grouped[-1].append(x) else: grouped.append([x]) return grouped  Example Usage: >>> lst = [1, 2, 3, 4, 5, 12, 13, 14, 15, 20, 21, 22, 23, 30, 35, 36, 37, 38, 39, 40] >>> group_by_missing(lst) [[1, 2, 3, 4, 5], [12, 13, 14, 15], [20, 21, 22, 23], [30], [35, 36, 37, 38, 39, 40]]
Question: I've a project in django 3.2 and I've updated (pip install -r requirements.txt) to version 4.0 (new release) and I've currently the below error when I run the server in a virtual environment. I use DRF. Can't import => from rest_framework import routers in urls.py from django.conf.urls import url ImportError: cannot import name 'url' from 'django.conf.urls'

Answer: As per the Django docs here:  url(regex, view, kwargs=None, name=None)¶ This function is an alias to django.urls.re_path().   Deprecated since version 3.1: Alias of django.urls.re_path() for backwards compatibility.  django.conf.urls.url() was deprecated since Django 3.1, and as per the release notes here, is removed in Django 4.0+. You can resolve the issue using path or re_path. https://docs.djangoproject.com/en/4.0/ref/urls/
Question: I have following statement in my code: mcap_summary['cap'].cat.set_categories(['Large','Mid','Small','None'],inplace=True)  Which now generates a warning as: D:\Python\Python39\lib\site-packages\pandas\core\arrays\categorical.py:2630: FutureWarning: The inplace parameter in pandas.Categorical.set_categories is deprecated and will be removed in a future version. Removing unused categories will always return a new Categorical object. res = method(*args, **kwargs) How am I suppose to write to avoid this warning and future errors? Thanks in advance

Answer: pandas v1.3.0+ deprecated the inplace option.  Deprecated the inplace parameter of Categorical.remove_categories(), Categorical.add_categories(), Categorical.reorder_categories(), Categorical.rename_categories(), Categorical.set_categories() and will be removed in a future version (GH37643) — https://pandas.pydata.org/pandas-docs/version/1.3.0/whatsnew/v1.3.0.html#other-deprecations  so you can code like below: mcap_summary['cap'] = mcap_summary['cap'].cat.set_categories(['Large', 'Mid', 'Small', 'None'])
Question: Is it possible to execute statements while the debug mode is active, possibly in the interactive mode? Let's say I'm working with a dataframe, and it doesn't behave as I want. I go line by line in debug mode, and I want to check some properties while doing that, for example the number of NaN values. Using the variable window to check the entries is obviously a waste of time for such a task. I could use write a print statement into my code, stop the debugging, start the debugging again, and then the print statement is part of the code and will be executed. But that would mean always stopping and starting debug, as soon as I run into an unforeseen problem and trying to find out what's happening. Is it possible to execute statements while still remaining in debug, at the line you currently are at. It would be especially good if there is a solution with running the statement in Interactive Mode because then I wouldn't need to mess with the original program. I hope it's clear but if not, I can try to construct an example with screenshots. Any help would be appreciated!

Answer: When on a breakpoint you can use the Debug Console to run Python code in the current context. It's in the same tab as "Problems", "Output" and "Terminal", typically under the Editor pane. See in the upper menu "View > Debug Console".  More info is available in the official Visual Studio Code documentation:  Debug Console REPL
Question: Pydantic model for compulsory field with alias is created as follows class MedicalFolderUpdate(RWModel): id : str = Field(alias='_id') university : Optional[str]  How to add optional field university's alias name 'school' as like of id?

Answer: It is not documented on the Pydantic website how to use the typing Optional with the Fields Default besides their allowed types in which they include the mentioned Optional:  Optional[x] is simply shorthand for Union[x, None]; see Unions below for more detail on parsing and validation and Required Fields for details about required fields that can receive None as a value.  for that, you would have to use their field customizations as in the example: class Figure(BaseModel): name: str = Field(alias='Name') edges: str = Field(default=None, alias='Edges')  without the default value, it breaks because the optional does not override that the field is required and needs a default value. Which is the solution I used to overcome this problem while using Pydantic with fast API to manage mongo resources
Question: In some data I am processing I am encountering data of the type float, which are filled with 'nan', i.e. float('nan'). However checking for it does not work as expected: float('nan') == float('nan') >> False  You can check it with math.isnan, but as my data also contains strings (For example: 'nan', but also other user input), it is not that convenient: import math math.isnan(float('nan')) >> True math.isnan('nan') >> TypeError: must be real number, not str  In the ideal world I would like to check if a value is in a list of all possible NaN values, for example: import numpy as np if x in ['nan', np.nan, ... ]: # Do something pass  Now the question: How can I still use this approach but also check for float('nan') values? And why equals float('nan') == float('nan') False

Answer: Why not just wrap whatever you pass to math.isnan with another float conversion? If it was already a float (i.e. float('nan')) you just made a "redundant" call: import math def is_nan(value): return math.isnan(float(value))  And this seems to give your expected results: >>> is_nan(float('nan')) True >>> is_nan('nan') True >>> is_nan(np.nan) True >>> is_nan(5) False >>> is_nan('5') False   This will still raise a ValueError for non-numeric (except 'nan') strings. If that's a problem, you can wrap with try/except. As long as the float conversion worked, there is no reason for isnan to fail. So we are basically catching non-numeric strings that my fail the float conversion: def is_nan(value): try: return math.isnan(float(value)) except ValueError: return False  Any non-numeric string is surely not a NaN value so return False.
Question: Is there any way I can use ccxt to extract the price of a crypto currency at a given time in the past? Example: get price of BTC on binance at time 2018-01-24 11:20:01

Answer: You can use the fetch_ohlcv method on the binance class in CCXT def fetch_ohlcv(self, symbol, timeframe='1m', since=None, limit=None, params={}):  You'll need the date as a timestamp in milliseconds, and you can only get it precise to the minute, so take away the seconds, or you'll get the price for the minute after timestamp = int(datetime.datetime.strptime("2018-01-24 11:20:00", "%Y-%m-%d %H:%M:%S").timestamp() * 1000)   You can only get the price of BTC in comparison to another currency, we'll use USDT(closely matches USD) as our comparison currency, so we will look up the price of BTC in the BTC/USDT market When we use the method, we will set since to your timestamp, but set the limit as one so that we only get one price import ccxt from pprint import pprint print('CCXT Version:', ccxt.__version__) exchange = ccxt.binance() timestamp = int(datetime.datetime.strptime("2018-01-24 11:20:00+00:00", "%Y-%m-%d %H:%M:%S%z").timestamp() * 1000) response = exchange.fetch_ohlcv('BTC/USDT', '1m', timestamp, 1) pprint(response)  Which will return candlestick values for one candle [ 1516792860000, // timestamp 11110, // value at beginning of minute, so the value at exactly "2018-01-24 11:20:00" 11110.29, // highest value between "2018-01-24 11:20:00" and "2018-01-24 11:20:59" 11050.91, // lowest value between "2018-01-24 11:20:00" and "2018-01-24 11:20:59" 11052.27, // value just before "2018-01-24 11:21:00" 39.882601 // The volume traded during this minute ]
Question: Consider the following code sample def sum(a: int, b: int): return a + b def wrap(*args, **kwargs): # delegate to sum return sum(*args, **kwargs)  The code works well except that type hint is lost. It's very common in Python to use *args, **kwargs to implement delegation pattern. It would be great to have a way to keep the type hint while using them, but I don't know if it is possible and how.

Answer: See https://github.com/python/typing/issues/270 for a long discussion of this problem. You can achieve this by decorating wrap with an appropriately typed identity function: F = TypeVar("F", bound=Callable) def copy_signature(_: F) -> Callable[..., F]: return lambda f: f def s(x: int, y: int) -> int: return x + y @copy_signature(s) def wrap(*args, **kwargs): s(*args, **kwargs) reveal_type(wrap) # Revealed type is "def (x: int, y: int) -> int"  As far as I know, the decorator is necessary - it is still not possible to do this using type hints alone, even with PEP612. Since it was already good practice to use the functools.wraps decorator in this situation (which copies the runtime type information), this is not such a loss - you could instead define def wraps(f: F) -> Callable[..., F]: return functools.wraps(f) # type: ignore  and then both the runtime and static type information should be correct so long as you use this decorator. (Sadly the typeshed stubs for functools.wraps included with mypy aren't quite restrictive enough to get this working out of the box.) PEP612 adds the ability to add/remove arguments in your wrapper (by combining ParamSpec with Concatenate), but it doesn't remove the need for some kind of higher-order function (like a decorator) to let the type system infer the signature of wrap from that of s.
Question: When I highlight a variable within a cell in colab, all the instances of that variable shines up.  Is it possible to change all these instances of the variables at the same time and if such, how?

Answer: Google colab uses the same keys short-cuts of microsoft VS, just use Ctrl+D to add to the selection the next occurrence. Use instead Ctrl+Shift+L to select all occurrence. Once you selected the occurrences you can edit them by just typing the new name. If you use short variable names ( as X) this method will probably not work, as every word with an x in it will also change.
Question: After upgrading to Django 4.0, I get the following error when running python manage.py runserver  ... File "/path/to/myproject/myproject/urls.py", line 16, in <module> from django.conf.urls import url ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py)  My urls.py is as follows: from django.conf.urls from myapp.views import home urlpatterns = [ url(r'^$', home, name="home"), url(r'^myapp/', include('myapp.urls'), ] 

Answer: django.conf.urls.url() was deprecated in Django 3.0, and is removed in Django 4.0+. The easiest fix is to replace url() with re_path(). re_path uses regexes like url, so you only have to update the import and replace url with re_path. from django.urls import include, re_path from myapp.views import home urlpatterns = [ re_path(r'^$', home, name='home'), re_path(r'^myapp/', include('myapp.urls'), ]  Alternatively, you could switch to using path. path() does not use regexes, so you'll have to update your URL patterns if you switch to path. from django.urls import include, path from myapp.views import home urlpatterns = [ path('', home, name='home'), path('myapp/', include('myapp.urls'), ]  If you have a large project with many URL patterns to update, you may find the django-upgrade library useful to update your urls.py files.
Question: I run FastAPI at PyCharm IDE and it always run 3 workers. I don't know why, but, the last instance created is being accessed on every API call. Could anyone can help how can I get single running worker? Code: import uvicorn from fastapi import FastAPI from fastapi.templating import Jinja2Templates from starlette.middleware.cors import CORSMiddleware app = FastAPI() app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]) print(f"main.py with :{app}") @app.get('/') def home(): return "Hello" if __name__ == "__main__": uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=False, log_level="debug", debug=True, workers=1, limit_concurrency=1, limit_max_requests=1)  Console output:  /Users/user/.pyenv/versions/3.7.10/bin/python /Users/user/github/my-project/backend/main.py main.py with :<fastapi.applications.FastAPI object at 0x102b35d50> INFO: Will watch for changes in these directories: ['/Users/user/github/my-project/backend'] INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: Started reloader process [96259] using statreload main.py with :<fastapi.applications.FastAPI object at 0x10daadf50> main.py with :<fastapi.applications.FastAPI object at 0x1106bfe50> INFO: Started server process [96261] INFO: Waiting for application startup. INFO: Application startup complete.  

Answer: Thanks to your last comment I understood better your question. Your real question So in fact your question is why is FastAPI object is created 3 times. In the log one can indeed see that you have 3 different memory adresses 0x102b35d50, 0x10daadf50, 0x1106bfe50 This doesn't mean that you have 3 workers, just that the FastAPI object is created 3 times. The last one this the one that your API will use. Why it happens The object is created multiple times, because :  First, you run main.py that go through the all code (one creation of FastAPI object), and then reach the __main__ Then uvicorn launch main:app so it go once again to the file main.py and build another FastAPI object. The last one is created by the debug=True when you set it to False you have one less FastAPI object created. I'm not quite sure why.  The solution The solution is to separate the API definition from the start of the API. For example, one could create a run.py file with : import uvicorn if __name__ == "__main__": uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=False, log_level="debug", debug=True, workers=1, limit_concurrency=1, limit_max_requests=1)  and launch this file. Another option would be to launch your API in command line: uvicorn main:app --host=0.0.0.0 --port=8000 --log-level=debug --limit-max-requests=1 --limit-concurrency=1  You can find all the command line arguments here
Question: I have two python packages. One is a util library, and one is an application that will use the util library (eventually I will have more apps sharing the library. I am using poetry for both, and the app specifies the common library as a dependency using the path and develop properties. For example, my layout looks something like this: - common/ - common/ - __init__.py - py.typed - pyproject.toml - myapp/ - myapp/ - __init__.py - py.typed - pyproject.toml  And the myapp\pyproject.toml looks something like this: [tool.poetry] name = "myapp" version = "0.1.0" description = "" authors = ["Your Name <you@example.com>"] [tool.poetry.dependencies] python = "^3.9" common = { path = "../common", develop = true } [tool.poetry.dev-dependencies] mypy = "^0.910" flake8 = "^4.0.1" black = {version = "^21.12b0", allow-prereleases = true} pytest = "^6.2.5" pytest-cov = "^3.0.0" pytest-mock = "^3.6.1" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"  When I run mypy on myapp I get something like: myapp/__init__.py:1:1: error: Cannot find implementation or library stub for module named "common" [import] 

Answer: Assuming the "util" package has type hints, you'll want to add a py.typed file to it (in the root directory of the package) so mypy understands it comes with type hints. py.typed should be empty, it's just a flag file. If your package does not have type hints, then you'd have to add stubs files (.pyi). More details: https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages
Question: when i add tensoflow with poetry (poetry add tensorflow) i get this error :  Using version ^2.7.0 for tensorflow Updating dependencies Resolving dependencies... (0.8s) SolverProblemError The current project's Python requirement (>=3.6,<4.0) is not compatible with some of the required packages Python requirement: - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0 - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0 - tensorflow-io-gcs-filesystem requires Python >=3.6, <3.10, so it will not be satisfied for Python >=3.10,<4.0 .... For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.6,<3.10" For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.6,<3.10" For tensorflow-io-gcs-filesystem, a possible solution would be to set the `python` property to ">=3.6,<3.10" 

Answer: The error message tells you, that your project aims to be compatible for python >=3.6,<4.0 (You probably have ^3.6 in your pyproject.toml), but pytorch says it's compatible only for >=3.6, <3.10. This is only a subset of the range of you project definition. Poetry doesn't care about your current environment. It cares about a valid project definition at all. The solution is already suggested within the error message. Set the version range for python in your pyproject.toml to >=3.6, <3.10.
Question: import pandas as pd df = pd.DataFrame({ "col1" : ["a", "b", "c"], "col2" : [[1,2,3], [4,5,6,7], [8,9,10,11,12]] }) df.to_parquet("./df_as_pq.parquet") df = pd.read_parquet("./df_as_pq.parquet") [type(val) for val in df["col2"].tolist()]  Output: [<class 'numpy.ndarray'>, <class 'numpy.ndarray'>, <class 'numpy.ndarray'>]  Is there any way in which I can read the parquet file and get the list values as pythonic lists (just like at creation)? Preferably using pandas but willing to try alternatives. The problem I'm facing is that I have no prior knowledge which columns hold lists, so I check types similarly to what I do in the code. Assuming I'm not interested in adding numpy currently as a dependency, is there any way to check if a variable is array-like without explicitly importing and specifying np.ndarray?

Answer: You can't change this behavior in the API, either when loading the parquet file into an arrow table or converting the arrow table to pandas. But you can write your own function that would look at the schema of the arrow table and convert every list field to a python list import pyarrow as pa import pyarrow.parquet as pq def load_as_list(file): table = pq.read_table(file) df = table.to_pandas() for field in table.schema: if pa.types.is_list(field.type): df[field.name] = df[field.name].apply(list) return df load_as_list("./df_as_pq.parquet")
Question: THE PROBLEM: In Discord, as you may know, there is an "About me" section. This section is a description of a profile that you can write yourself. Bots can have an "About me" section. What I want to do is to edit this "About me" section automatically in discord.py; For example, the "About me" section of the bot changes every hour. WHAT I TRIED: I searched for some answers for a long time but didn't find anything relevant. I saw that you could modify the "about me" with the Developer portal, but it's not automated. I saw that some people said "This will be able in discord.py V2" but didn't find it It may be possible to resolve this problem with HTTP requests but it's only a supposition, I'm not very good at this topic. CODE: @bot.event async def on_ready(): while 1: await change_the_about_me_section(str(random.randint(0,1000)) time.sleep(3600) # change_the_about_me_section isn't a real function # I just wanted to show an exemple of what I wanted to do 

Answer: There is an answer. You can fully automate it with Python in a few lines. With the requests library. You just have to first include requests with: import requests  Then, do a patch request requests.patch(url="https://discord.com/api/v9/users/@me", headers= {"authorization": token}, json = {"bio": abio} ) # With token your token, and abio a string like "hello"  And... that's it :) (Note: You can also do that with the color of the Account with accent_color instead of bio, and the banner with banner instead of bio)
Question: The code import logging from tenacity import retry, wait_incrementing, stop_after_attempt import tenacity @retry(wait=wait_incrementing(start=10, increment=10, max=100), stop=stop_after_attempt(3)) def print_msg(): logging.info('Hello') logging.info("World") raise Exception('Test error') if __name__ == '__main__': logging.basicConfig( format='%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s', datefmt='%d-%m-%Y:%H:%M:%S', level=logging.INFO) logging.info('Starting') print_msg()  output 21-11-2018:12:40:48,586 INFO [retrier.py:18] Starting 21-11-2018:12:40:48,586 INFO [retrier.py:8] Hello 21-11-2018:12:40:48,586 INFO [retrier.py:9] World 21-11-2018:12:40:58,592 INFO [retrier.py:8] Hello 21-11-2018:12:40:58,592 INFO [retrier.py:9] World 21-11-2018:12:41:18,596 INFO [retrier.py:8] Hello 21-11-2018:12:41:18,596 INFO [retrier.py:9] World 21-11-2018:12:41:18,596 ERROR [retrier.py:22] Received Exception ....  How to log it's retrying? Such as Error. Retrying 1... ... Error. Retrying 2... ... 

Answer: You can write your own callback function to get attempt_number from retry_state. code: def log_attempt_number(retry_state): """return the result of the last call attempt""" logging.error(f"Retrying: {retry_state.attempt_number}...")  log the attempt after every call of the function as shown below @retry(wait=wait_incrementing(start=10, increment=10, max=100), stop=stop_after_attempt(3), after=log_attempt_number)   Output: 14-12-2021:19:01:26,716 INFO [<ipython-input-15-f6916dbe7ec1>:24] Starting 14-12-2021:19:01:26,718 INFO [<ipython-input-15-f6916dbe7ec1>:15] Hello 14-12-2021:19:01:26,720 INFO [<ipython-input-15-f6916dbe7ec1>:16] World 14-12-2021:19:01:26,723 ERROR [<ipython-input-15-f6916dbe7ec1>:11] Retrying: 1... 14-12-2021:19:01:36,731 INFO [<ipython-input-15-f6916dbe7ec1>:15] Hello 14-12-2021:19:01:36,733 INFO [<ipython-input-15-f6916dbe7ec1>:16] World 14-12-2021:19:01:36,735 ERROR [<ipython-input-15-f6916dbe7ec1>:11] Retrying: 2... 14-12-2021:19:01:56,756 INFO [<ipython-input-15-f6916dbe7ec1>:15] Hello 14-12-2021:19:01:56,758 INFO [<ipython-input-15-f6916dbe7ec1>:16] World 14-12-2021:19:01:56,759 ERROR [<ipython-input-15-f6916dbe7ec1>:11] Retrying: 3...