eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have just started learning python. I am working on netflix_tiles dataset downloaded from Kaggle. Some of the entries in the director column have multiple director names separated by column, i was trying to separate director names using the split function The following is one of the original values loaded from the file to dataframe  s7 Movie My Little Pony: A New Generation, Robert Cullen, José Luis Ucha Vanessa Hudgens, ..  I am using the following code to do the split def strip(x): x = x.strip().split(',') return x director_counts = df["director"].apply(strip)  After the above code executes the output is as follows  s7 [Robert Cullen, José Luis Ucha]  The director name is not split based on comma and I am also seeing the index(s7) is also returned from the function when i passed just the director column to the function. Can anyone please tell me why is it behaving in this way? Edit: Tried this as well director_counts = df['director'].str.split(',\s*')  Link to collab: https://colab.research.google.com/drive/1OXJ9XKCBVg4-6W8Hiqfy4ZTkgz0IVqbR?usp=sharing

Answer: Use str.strip(): df = pd.read_csv('/home/damien/Downloads/netflix_titles.csv.zip') directors = df['director'].str.split(',\s*')  Output: >>> directors 0 [Kirsten Johnson] 1 NaN 2 [Julien Leclercq] 3 NaN 4 NaN ... 8802 [David Fincher] 8803 NaN 8804 [Ruben Fleischer] 8805 [Peter Hewitt] 8806 [Mozez Singh] Name: director, Length: 8807, dtype: object  Update  I was expecting it to come in two different rows  Use explode: >>> directors.explode() 0 Kirsten Johnson 1 NaN 2 Julien Leclercq 3 NaN 4 NaN ... 8802 David Fincher 8803 NaN 8804 Ruben Fleischer 8805 Peter Hewitt 8806 Mozez Singh Name: director, Length: 9612, dtype: object # <- 9612 rows instead of 8807  To get count per director, use value_counts (drop nan by default): >>> directors.explode().value_counts() Rajiv Chilaka 22 Jan Suter 21 Raúl Campos 19 Suhas Kadav 16 Marcus Raboy 16 .. Raymie Muzquiz 1 Stu Livingston 1 Joe Menendez 1 Eric Bross 1 Mozez Singh 1 Name: director, Length: 4993, dtype: int64
Question: How do I access the value of an object attribute that displays in angle brackets like: <attribute 'version' of 'Metashape.Metashape.Application' objects>? Specifically, I am using the Metashape Python module and run the following lines within an interactive Python session: import Metashape a = Metashape.Application a.version  and this is when I get <attribute 'version' of 'Metashape.Metashape.Application' objects>  I've tried print(a.version) and get the same output. According to the module reference doc, this attribute should be a string, so I'm confused why it can't just be displayed as a string.

Answer: According to the docs:  An instance of Application object can be accessed using Metashape.app attribute, so there is usually no need to create additional instances in the user code.  so... import Metashape print(Metashape.app.version)  If you want to do it your way, you need to instance Application. import Metashape app = Metashape.Application() print(app.version)
Question: I'm trying to get content of website but my requests return me an 403 ERROR. After searching, I found Network>Headers section to add headers before GET request and tried these headers. from bs4 import BeautifulSoup as bs import requests url = "https://clutch.co/us/agencies/digital-marketing" HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"} ### Also tried "Referer" , "sec-ch-ua-platform" and "Origin" headers but nothing changed. html = requests.get(url,headers=HEADERS) print("RESULT:",html)  But result didn't change.

Answer: You can try to load the page from the Google cache instead directly: import requests from bs4 import BeautifulSoup url = "https://clutch.co/us/agencies/digital-marketing" cache_URL = "https://webcache.googleusercontent.com/search?q=cache:" def get_data(link): hdr = { "User-Agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Mobile Safari/537.36" } req = requests.get(cache_URL + link, headers=hdr) content = req.text return content soup = BeautifulSoup(get_data(url), 'html.parser') for h3 in soup.select('h3.company_info'): print(h3.get_text(strip=True))  Prints: WebFX Ignite Visibility SmartSites Thrive Internet Marketing Agency Lilo Social NEWMEDIA.COM Funnel Boost Media Direct Online Marketing SeedX Inc. Impactable ...
Question: I have previously arranged a Python repository without a src folder, and got it running with: pdm install --dev pdm run mymodule  I am failing to replicate the process in a repository with a src folder. How do I do it? pyproject.toml [project] name = "mymodule" version = "0.1.0" description = "Minimal Python repository with a src layout." requires-python = ">=3.10" [build-system] requires = ["pdm-pep517>=1.0.0"] build-backend = "pdm.pep517.api" [project.scripts] mymodule = "cli:invoke"  src/mymodule/__init__.py Empty file. src/mymodule/cli.py def invoke(): print("Hello world!") if __name__ == "__main__": invoke()  With the configuration above, I can pdm install --dev but pdm run mymodule fails with: Traceback (most recent call last): File "/home/user/Documents/mymodule/.venv/bin/mymodule", line 5, in <module> from cli import invoke ModuleNotFoundError: No module named 'cli' 

Answer: You need to modify your pyproject.toml as follows: [project.scripts] mymodule = "mymodule.cli:invoke"  For good measure, you may want to delete the .venv folder and .pdm.toml file before running pdm install --dev again.
Question: i have about 1300 CSV files with almost 40k of rows in each file, i have written a python code to read the file and convert all the 40k entries into single insert statement to insert in Postgres database. The psudocode is following for file in tqdm(files, desc="Processing files"): rows = ReadFile(file) # read all 40k rows from the file q = GenerateQuery(rows) # convert all rows into single bulk insert statement InsertData(q) # Execute the generated query to insert data  The code is fine but there are performance issues, when I start the code with empty table it takes around 2 to 3it/s, but after 15 to 20 files it takes 10 to 12it/s, and then the performance drops exponentially with each 10 to 15 files processing, the per iteration time keeps on increasing even it reaches 40 to 50s/it, it's hilarious, according to my understanding I have developed the following hypothesis Since in start table is empty its very easy to update table indexes, so it takes no time for 40k bulk insert records to update indexes but with growing records it become harder and even harder to update indexes in the table with 10m+ records. My Question is can I temporarily disable index updation of the table, so that after complete data dump I will then manually update the indexes by calling some query in postgres which for now I don't know if it really exists.

Answer: No, you cannot disable indexes. If the table is empty or almost empty when you start loading, you can win by dropping the indexes before you start and creating them again afterwards. However, INSERT performance should stay constant over time. It is difficult to guess what might be the cause. You can try using auto_explain to capture EXPLAIN (ANALYZE, BUFFERS) output for a slow insert, that may give you a clue. For bulk loads, you will get the best performance with COPY.
Question: How to replace 422 standard exception with custom exception only for one route in FastAPI? I don't want to replace for the application project, just for one route. I read many docs, and I don't understand how to do this. Example of route that I need to change the 422 exception: from fastapi import APIRouter from pydantic import BaseModel router = APIRouter() class PayloadSchema(BaseModel): value_int: int value_str: str @router.post('/custom') async def custom_route(payload: PayloadSchema): return payload 

Answer: You can register multiple error handlers with the router. You can re-declare the default handler, and then optionally call it depending on the path: class PayloadSchema(BaseModel): value_int: int value_str: str router = APIRouter() @router.post('/standard') async def standard_route(payload: PayloadSchema): return payload @app.exception_handler(RequestValidationError) async def standard_validation_exception_handler(request: Request, exc: RequestValidationError): return JSONResponse( status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content=jsonable_encoder({"detail": exc.errors(), "body": exc.body}), ) @router.post('/custom') async def custom_route(payload: PayloadSchema): return payload @app.exception_handler(RequestValidationError) async def custom_exception_handler(request: Request, exc: RequestValidationError): if (request.url.path == '/custom'): return JSONResponse({"error": "Bad request, must be a valid PayloadSchema format"}, status_code=400) else: return await standard_validation_exception_handler(request, exc) app = FastAPI() app.include_router(router) app.add_exception_handler(RequestValidationError, custom_exception_handler)  Or you can do it directly with the @app.exception_handler decorator without the router (see FastAPI docs).
Question: I am looking to find a solution to check multiple conditions within a group. First I'm checking for overlaps between records (based on id) and second I should make an exception for the highest numbered record within the same contagious overlap. On top of that, there can be multiple overlaps for same id. The example: data = [('A',1000,1,100), ('B',1001,0,10), ('B',1002,10,15), ('B',1002,20,22), ('B',1003,25,50), ('B',1004,50,55), ('B',1005,53,56), ('B',1006,60,100), ('C',1007,1,100) ] schema = StructType([ \ StructField("id",StringType(),True), \ StructField("tran",IntegerType(),True), \ StructField("start",IntegerType(),True), \ StructField("end",IntegerType(),True), \ ]) df = spark.createDataFrame(data=data,schema=schema) df.show() +---+----+-----+---+ | id|tran|start|end| +---+----+-----+---+ | A|1000| 1|100| | B|1001| 0| 10| | B|1002| 10| 15| | B|1003| 20| 22| | B|1004| 25| 50| | B|1005| 50| 55| | B|1006| 53| 56| | B|1007| 60|100| | C|1008| 1|100| +---+----+-----+---+  The desired dataframe should look like this: | id|tran|start|end|valid| +---+----+-----+---+-----+ | A|1000| 1|100| yes| # this is valid because by id there is no overlap between start and end | B|1001| 0| 10| no| # invalid because by id it overlaps with the next | B|1002| 10| 15| yes| # it overlaps with the previous one but it has the highest tran number between the two | B|1003| 20| 22| yes| # yes because no overlap | B|1004| 25| 50| no| # invalid because overlaps and the tran is not the highest | B|1005| 50| 55| no| # invalid because overlaps and the tran is not the highest | B|1006| 53| 56| yes| # it overlaps with the previous ones but it has the highest tran number among the three contagiously overlapping ones | B|1007| 60|100| yes| # no overlap | C|1008| 1|100| yes| # no overlap +---+----+-----+---+-----+  Massive thanks for the Legend who solves this :)

Answer: Join with self, and group by for the dataframe, count how many of the records are overlapped and get the validation. df.alias('a').join( df.alias('b'), (f.col('a.id') == f.col('b.id')) & (f.col('a.start') != f.col('b.start')) & (f.col('b.start').between(f.col('a.start'), f.col('a.end'))), 'left' ) \ .groupBy('a.id', 'a.tran', 'a.start', 'a.end') \ .agg(f.count('b.id').alias('valid')) \ .withColumn('valid', f.when(f.col('valid') == f.lit(0), 'yes').otherwise('no')) \ .show() +---+----+-----+---+-----+ | id|tran|start|end|valid| +---+----+-----+---+-----+ | B|1001| 0| 10| no| | A|1000| 1|100| yes| | B|1002| 10| 15| yes| | B|1002| 20| 22| yes| | B|1003| 25| 50| no| | B|1004| 50| 55| no| | B|1005| 53| 56| yes| | B|1006| 60|100| yes| | C|1007| 1|100| yes| +---+----+-----+---+-----+
Question: Using pydantic setting management, how can I load env variables on nested setting objects on a main settings class? In the code below, the sub_field env variable field doesn't get loaded. field_one and field_two load fine. How can I load an environment file so the values are propagated down to the nested sub_settings object? from typing import Optional from pydantic import BaseSettings, Field class SubSettings(BaseSettings): sub_field: Optional[str] = Field(None, env='SUB_FIELD') class Settings(BaseSettings): field_one: Optional[str] = Field(None, env='FIELD_ONE') field_two: Optional[int] = Field(None, env='FIELD_TWO') sub_settings: SubSettings = SubSettings() settings = Settings(_env_file='local.env') 

Answer: There are some examples of nested loading of pydantic env variables in the docs. Option 1 If you're willing to adjust your variable names, one strategy is to use env_nested_delimiter to denote nested fields. This appears to be the way that pydantic expects nested settings to be loaded, so it should be preferred when possible. So with a local.env like this: FIELD_ONE=one FIELD_TWO=2 SUB_SETTINGS__SUB_FIELD=value  You should be able to load the settings in this way from typing import Optional from pydantic import BaseModel, BaseSettings class SubSettings(BaseModel): # ^ Note that this inherits from BaseModel, not BaseSettings sub_field: Optional[str] class Settings(BaseSettings): field_one: Optional[str] field_two: Optional[int] sub_settings: SubSettings class Config: env_nested_delimiter = '__'  Alternate If you don't want to use the env_nested_delimiter functionality, you could load both sets of settings from the same local.env file. Then pass the loaded SubSettings to Settings directly. This can be done by overriding the Settings class __init__ method With this local.env FIELD_ONE=one FIELD_TWO=2 SUB_FIELD=value  use the following to load the settings from typing import Optional from pydantic import BaseModel, BaseSettings, Field class SubSettings(BaseSettings): sub_field: Optional[str] class Settings(BaseSettings): field_one: Optional[str] field_two: Optional[int] sub_settings: SubSettings def __init__(self, *args, **kwargs): kwargs['sub_settings'] = SubSettings(_env_file=kwargs['_env_file']) super().__init__(*args, **kwargs) settings = Settings(_env_file='local.env')
Question: I m trying to build a telegram bot usin the telegram.ext.Application class of python-telegram-bot and I need the bot to send periodic messages to a specific user while also running its polling loop to receive incoming messages from other users. I ve done a lot-lot-lot of searching and tried various combinations but I cant seem to get it to work. THis is the code I have so far (I removed message handlers to give you the minimum possible code example) import asyncio import logging from telegram.ext import Application token = '1111111111:AAAoOOV_Aaa1aaAAAaA1AA11A1aAaAAaAAA' user_id = 0000000000 application = Application.builder().token(token).build() async def send_message_to_user(user_id, message_text): try: await application.bot.send_message(chat_id=user_id, text=message_text) except Exception as e: logging.error(f"Error while sending message to user {user_id}: {e}") async def main() -> None: await send_message_to_user(user_id, "hello user, dont forget to get your pills") application.run_polling() if __name__ == "__main__": asyncio.run(main())  The send_message_to_user coroutine works fine on its own, but when I try to combine it with the run_polling method (with or without await before it), i get this error RuntimeError("Cannot close a running event loop") RuntimeError: Cannot close a running event loop sys:1: RuntimeWarning: coroutine 'Application.shutdown' was never awaited sys:1: RuntimeWarning: coroutine 'Application.initialize' was never awaited  I think run_polling has its own event loop and cant run asyncronously but unfortunately Im not experienced with asynchronous programming so can anyone help me figure out how to send periodic messages to a specific user while also running a telegram bot's polling loop? Thanks in advance for your help!

Answer: python-telegram-bot comes with the JobQueue, a built-in scheduling interface that allows you to run repeating tasks seamlessly within the framework. You might also be interested in this wiki section, which covers the general use case of running some asyncio logic in parallel to the bot.  Disclaimer: I'm currently the maintainer of python-telegram-bot.
Question: I have millions of compressed .bz2 files which I need to uncompressed. Can uncompression be parallelized ? I have access to the server with many cpu cores for the purpose. I worked with the following code which is correct but it is extremely slow. import os, glob, bz2 files = glob.glob("/data01/*.bz2") for fi in files: fo = fi[:-4] with bz2.BZ2File(fi) as fr, open(fo, "wb") as fw: shutil.copyfileobj(fr, fw) 

Answer: Multithreading would ideal for this because it's primarily IO-bound. from concurrent.futures import ThreadPoolExecutor import glob import bz2 import shutil def process(filename): with bz2.BZ2File(filename) as fr, open(filename[:-4], "wb") as fw: shutil.copyfileobj(fr, fw) def main(): with ThreadPoolExecutor() as tpe: tpe.map(process, glob.glob('/data01/*.bz2')) if __name__ == '__main__': main()
Question: I want to create a raster file (.tif) from a points file using a geopandas.geodataframe.GeoDataFrame object. My dataframe has two columns: [geometry] and [Value]. The goal is to make a 10m resolution raster in [geometry] point with the [Value] value. My dataset is: geometry | Value 0 | POINT (520595.000 5720335.000) | 536.678345 1 | POINT (520605.000 5720335.000) | 637.052185 2 | POINT (520615.000 5720335.000) | 1230.553955 3 | POINT (520625.000 5720335.000) | 944.970642 4 | POINT (520635.000 5720335.000) | 1094.613281 5 | POINT (520645.000 5720335.000) | 1123.185181 6 | POINT (520655.000 5720335.000) | 849.37634 7 | POINT (520665.000 5720335.000) | 1333.459839 8 | POINT (520675.000 5720335.000) | 492.866608 9 | POINT (520685.000 5720335.000) | 960.957214 10 | POINT (520695.000 5720335.000) | 539.401978 11 | POINT (520705.000 5720335.000) | 573.015625 12 | POINT (520715.000 5720335.000) | 970.386536 13 | POINT (520725.000 5720335.000) | 390.315094 14 | POINT (520735.000 5720335.000) | 642.036865  I have tried before and so, I know that with from geocube.api.core import make_geocube I could do it, but due to some libraries I have a limitation and I cannot use make_geocube. Any idea?

Answer: Assign x and y columns, convert to xarray, then export to tiff using rioxarray: # do this before sending to xarray # to ensure extension is loaded import rioxarray # assuming your GeoDataFrame is called `gdf` gdf["x"] = gdf.x gdf["y"] = gdf.y da = ( gdf.set_index(["y", "x"]) .Value .to_xarray() ) da.rio.to_raster("myfile.tif")  In order for this to work, the points must make up a full regular grid, with values of x and y each repeated for each combination. If this is instead just a collection of arbitrary points converting to xarray with x and y as perpendicular dimensions will explode your memory and the result will be almost entirely NaNs.
Question: I am writing an application that should test the solution of many students. I have structure like this app/ students/ A/ lab1/ solution.py lab2 solution.py B/ lab1/ solution.py C/ test.py  I want to import the solution file in the testing module and run the main method. I know there is __import__ function that returns a module object, but I don't know how to call main method. My code: import os def get_student_names(): return os.listdir('students') def test(name, lab): if lab not in os.listdir('/'.join(['students',name])): return f"{lab} not found" if 'solution.py' not in os.listdir('/'.join(['students',name,lab])): return "solution file not found" module = __import__('.'.join(['students',name,lab,'solution'])) # module.name.lab.solution.main() def main(): student_names = get_student_names() for student in student_names: result = test(student, "lab1")  I want to call module.name.lab.solution.main() but the module is students. How to switch to solution submodule?

Answer: Your most immediate problem is that you aren't returning anything from test so, result will always be None. The second and most important problem you have is that you don't really have any modules. You actually have a bunch of random folders with arbitrary python scripts in them. This means __import__ is going to be quite unfriendly to you. if we run this line __import__('students.A.lab1.solution').main()  we get back this error: AttributeError: module 'students' has no attribute 'main'  students? We're supposed to be in solution! That's a problem. There is a way to do this without worrying about converting and maintaining your individual folders as actual modules.  First, students must call main in their solution. ex: def main(): print('hello, world') main()  Secondly, run their script. If the command fails it is going to tell you the script doesn't exist and move on to the next student. import os #change this to however you call python PY = 'python3' #lab format strings LAB1 = '{} "students/{}/lab1/solution.py"' LAB2 = '{} "students/{}/lab2/solution.py"' def main(lab:str) -> None: for student in os.listdir('students'): os.system(lab.format(PY, student)) main(LAB1)  As highlighted by @KarlKnetchel, the above code has all of the same vulnerabilities as your original concept. This is NOT safe. You should definitely consider running this in a sandbox, if you use it.
Question: I need to iterate over a copy of a set that doesn't contain a certain element. Until now I'm doing this: for element in myset: if element != myelement: ...  But I want something like this, in one line: for element in myset.copy().remove(myelement): ...  Obviously this doesn't work because the remove method returns None.

Answer: Use the set difference operator. for element in myset - {myelement}: ...  This creates a new set containing the elements of myset that aren't in {myelement} (namely, myelement itself).
Question: I want to calculate the centroid of a figure formed by the points: (0,0), (70,0), (70,25), (45, 45), (45, 180), (95, 188), (95, 200), (-25, 200), (-25,188), (25,180), (25,45), (0, 25), (0,0). I know that the correct result for the centroid of this polygon is x = 35 and y = 100.4615 (source), but the code below does not return the correct values (figure of the polygon below). import numpy as np points = np.array([(0,0), (70,0), (70,25), (45,45), (45,180), (95,188), (95,200), (-25,200), (-25, 188), (25,180), (25,45), (0,25), (0,0)]) centroid = np.mean(points, axis=0) print("Centroid:", centroid)  Output: Centroid: [32.30769231 98.15384615] How can I correctly calculate the centroid of the polygon? 

Answer: Fixed: def centroid(vertices): x, y = 0, 0 n = len(vertices) signed_area = 0 for i in range(len(vertices)): x0, y0 = vertices[i] x1, y1 = vertices[(i + 1) % n] # shoelace formula area = (x0 * y1) - (x1 * y0) signed_area += area x += (x0 + x1) * area y += (y0 + y1) * area signed_area *= 0.5 x /= 6 * signed_area y /= 6 * signed_area return x, y x, y = centroid(vertices) print(x, y)  Produces: 35.0 100.46145124716553
Question: The statement  I'm reading data sets using Polars.read_csv() method via a Python file handler:   with gzip.open(os.path.join(getParameters()['rdir'], dataset)) as compressed_file: df = pl.read_csv(compressed_file, sep = '\t', ignore_errors=True)   A performance warning keeps popping up:  Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.  Possible solutions  I already tried Python warning suppression, but it seems Polars literally just prints out this statement without any default warning associated. Another possibility would be to read using non-handler methods?  Any ideas on how to get rid of this annoying message will be highly appreciated.

Answer: I had a similar issue with opening from a ZipFile object. The solution was to add a .read() method to the filename. Maybe the same would work in your case?  with gzip.open(os.path.join(getParameters()['rdir'], dataset)) as compressed_file: df = pl.read_csv(compressed_file.read(), sep = '\t', ignore_errors=True)