eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following pandas data frame: the_df = pd.DataFrame({'id':[1,2],'name':['Joe','𝒮𝒶𝓇𝒶𝒽']}) the_df id name 0 1 Joe 1 2 𝒮𝒶𝓇𝒶𝒽  As you can see, we can read the second name as "Sarah", but it's written with special characters. I want to create a new column with these characters converted to latin characters. I have tried this approach: the_df['latin_name'] = the_df['name'].str.extract(r'(^[a-zA-Z\s]*)') the_df id name latin_name 0 1 Joe Joe 1 2 𝒮𝒶𝓇𝒶𝒽  But it doesn't recognize the letters. Please, any help on this will be greatly appreciated.

Answer: Try .str.normalize the_df['name'].str.normalize('NFKC').str.extract(r'(^[a-zA-Z\s]*)')  Output:  0 0 Joe 1 Sarah
Question: Here's the Python code I'm working on: def inver_hopf(x,y,z): return (1/np.sqrt(x**2+y**2+(1+z)**2))*np.matrix([[1+z],[x+y.j]],dtype=complex)  The problem happens at [x+y.j], where j means complex unit. It returns me the error message AttributeError: 'int' object has no attribute 'j'. If I remove the dot, then it returns NameError: name 'yj' is not defined. How can I correct that? Thanks!

Answer: j alone is a variable, you can have the complex number by typing 1j
Question: This is my code. It is everything I have in my programme: from pathlib import Path new_dir = Path.home() / "new_directory" file_path = new_dir / "program2.py" file_path.unlink(missing_ok=True)  The file program2.py does not exist; that is why I wanted to set the missing_ok parameter to True so that it would not raise a FileExistsError. But every time I run the code it gives me the following message: file_path.unlink(missing_ok=True) TypeError: unlink() got an unexpected keyword argument 'missing_ok' Do I have an outdated version of python or have I made a mistake in the code, help would be much appreciated!

Answer: The missing_ok parameter was added to Path.unlink only on python 3.8. You should upgrade python to newer version if you want to use this parameter. You can check your python version with the command python -V
Question: If I want to use database while processing a request, I make a Dependency Injection like this: @app.post("/sample_test") async def sample_test(db: Session = Depends(get_db)): return db.query(models.User.height).all()  But I cannot do it with events like this: @app.on_event("startup") async def sample_test(db: Session = Depends(get_db)): return db.query(models.User.height).all()  because starlette events don't support Depends. This is my get_db() function: def get_db(): db = SessionLocal() try: yield db finally: db.close()  just like in FastAPI manual (https://fastapi.tiangolo.com/tutorial/sql-databases/). How can I access get_db() inside my event function, so I can work with a Session? I've tried: @app.on_event("startup") async def sample_test(db: Session = Depends(get_db)): db = next(get_db()) return db.query(models.User.height).all()  but it doesn't work. I use MSSQL, if it's important.

Answer: Instead of using a dependency you can import the SessionLocal you've created as shown in the FastAPI manual and use a contextmanager to open and close this session: @app.on_event("startup") async def sample_test(): with SessionLocal() as db: return db.query(models.User.height).all()
Question: So, I want to convert UTC date time 2021-08-05 10:03:24.585Z to Indian date time how to convert it? What I tried is from datetime import datetime from pytz import timezone st = "2021-08-05 10:03:24.585Z" datetime_object = datetime.strptime(st, '%Y-%m-%d %H:%M:%S.%fZ') local_tz = timezone('Asia/Kolkata') start_date = local_tz.localize(datetime_object) print(start_date.replace(tzinfo=local_tz))  But Still the output is not with the timezone I have mentioned how can I convert the time and print the time from the time. Output: 2021-08-05 10:03:24.585000+05:21 

Answer: You can use sth like this: from datetime import datetime from dateutil import tz from_zone = tz.gettz('UTC') to_zone = tz.gettz('Asia/Kolkata') utc = datetime.strptime('2011-01-21 02:37:21', '%Y-%m-%d %H:%M:%S') utc = utc.replace(tzinfo=from_zone) central = utc.astimezone(to_zone)
Question: I have a list structure like this: listpost = [ { "post_id":"01", "text":"abc", "time": datetime.datetime(2021, 8, 5, 15, 53, 19), "type":"normal", }, { "post_id":"02", "text":"nothing", "time":datetime.datetime(2021, 8, 5, 15, 53, 19), "type":"normal", } ]  I want to filter the list by text in [text] key if only the [text] has "abc" so the example will look like this listpost = [ { "post_id":"01", "text":"abc", "time": datetime.datetime(2021, 8, 5, 15, 53, 19), "type":"normal", } ]  My code: from facebook_scraper import get_posts listposts = [] for post in get_posts("myhealthkkm", pages=1): listposts.append(post) print(listposts) 

Answer: Since you specifically asked about filtering the list you have, you can use filter builtin with lambda to filter out the elements from the list. >>> list(filter(lambda x: x.get('text', '')=='abc', listpost)) [{'post_id': '01', 'text': 'abc', 'time': datetime.datetime(2021, 8, 5, 15, 53, 19), 'type': 'normal'}]  But I'd recommend to filter it out upfront before actually appending it to the list, to avoid unnecessary computations due to the need to re-iterate the items i.e. appending only the items that match the criteria. Something like this: for post in get_posts("myhealthkkm", pages=1): if <post match the condition>: listposts.append(post) # append the post
Question: I'm trying to make a program that opens multiple websites, but I can't get it to press control-t. I've tried multiple solutions, but I can't find one that works. When I do the keydown method, I get an error that says webdriver has no attribute key_down  and when I try send_keys(Keys.CONTROL + 't') it doesn't raise any errors, or do anything. How can I open a new tab? Here's my try : from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.action_chains import ActionChains import time PATH = "C:\Program Files (x86)\chromedriver.exe" driver = webdriver.Chrome(PATH) driver.get("https://youtube.com") search = driver.find_element_by_id("search") #search.keydown(Keys.CONTROL) #Webelement.key_down(Keys.CONTROL).send_keys('t').key_up(Keys.CONTROL).perform() search.send_keys(Keys.CONTROL+'t') time.sleep(10) 

Answer: You can do it as from selenium import webdriver driver.get("https://www.youtube.com") search = driver.find_element_by_id("search") driver.execute_script("window.open('https://www.google.com')")
Question: I'm wondering whether it's possible to "freeze" a dataclass object in post_init() or even after an object was defined. So instead of: @dataclass(frozen=True) class ClassName: var1: type = value  Having something like: @dataclass class ClassName: var1: type = None def __post_init__(self): self.var1 = value FREEZE()  Or even sth like: a = ClassName() FREEZE(a)  Possible or not and why?

Answer: No, it isn't. But "frozen" can be subverted trivially, just use: @dataclass(frozen=True) class ClassName: var1: type = value def __post_init__(self): object.__setattr__(self, 'var1', value)
Question: This is probably a highly discussed topic, but i have not found "the answer" yet. I am inserting big tables into Azure SQL Server monthly. I process the raw data in memory with python and Pandas. I really like the speed and versatility of Pandas. Sample DataFrame size = 5.2 million rows, 50 columns, 250 MB memory allocated Transferring the processed Pandas DataFrame to Azure SQL Server is always the bottleneck. For data transfer, I used to_sql (with sqlalchemy). I tried fast_executemany, various chunk sizes etc arguments. The fastest way I found so far is to export the DataFrame to a csv file, then BULK INSERT that into SQL server using either SSMS, bcp, Azure Blob etc. However, i am looking into doing this bypassing the csv file creation, since my df has all the dtypes set, already loaded in memory. What is your fastest means of transfer this df to SQL Server, utilizing solely python/Pandas? I am also interested in solutions like using binary file transfer etc. - as long as I eliminate flat file export/import. Thanks

Answer: I had a similar issue, and I resolved it using a BCP utility. The basic description of the bottleneck issue is that it seems to be using RBAR data entry, as in Row-By-Agonizing-Row inserts, i.e. one insert statement/record. Going the bulk insert route has saved me a lot of time. The real benefit seemed to come once I crossed the threshold of 1M+ records, which you seem to well ahead of.  Link to utility:https://github.com/yehoshuadimarsky/bcpandas
Question: I need to create an endpoint that can receive the following JSON and recognize the objects contained in it: {​ "data": [ {​ "start": "A", "end": "B", "distance": 6 }​, {​ "start": "A", "end": "E", "distance": 4 }​ ] }  I created a model to handle a single object: class GraphBase(BaseModel): start: str end: str distance: int  And with it, I could save it in a database. But now I need to receive a list of objects and save them all. I tried to do something like this: class GraphList(BaseModel): data: Dict[str, List[GraphBase]] @app.post("/dummypath") async def get_body(data: schemas.GraphList): return data  But I keep getting this error on FastApi: Error getting request body: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) and this message on the response: { "detail": "There was an error parsing the body" }  I'm new to python and even newer to FastApi, how can I transform that JSON to a list of GraphBaseto save them in my db?

Answer: This is a working example. from typing import List from pydantic import BaseModel from fastapi import FastAPI app = FastAPI() class GraphBase(BaseModel): start: str end: str distance: int class GraphList(BaseModel): data: List[GraphBase] @app.post("/dummypath") async def get_body(data: GraphList): return data  I could try this API on the autogenerated docs.  Or, on the console (you may need to adjust the URL depending on your setting): curl -X 'POST' \ 'http://localhost:8000/dummypath' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "data": [ { "start": "string", "end": "string", "distance": 0 } ] }'  The error looks like the data problem. And I found that you have extra spaces at several places. Try the following: { "data": [ { "start": "A", "end": "B", "distance": 6 }, { "start": "A", "end": "E", "distance": 4 } ] }  The positions of extra spaces (which I removed) are below:
Question: >>arr = [4, 2, 1, 3] >>arr[0], arr[arr[0]-1] = arr[arr[0]-1], arr[0] >>arr  Result I expect >>[3, 2, 1, 4] Result I get >>[3, 2, 4, 3] Basically I'm trying to swap the #4 and #3 (In my actual problem, the index wont be 0, but rather an iterator "i" . So I cant just do arr[0], arr[3] = arr[3], arr[0]) I thought I understood simultaneous assignment fairly well. Apparently I was mistaken. I don't understand why arr[arr[0]-1] on the left side of the assignment is evaluating to arr[2] instead of arr[3]. If the assignments happen simultaneously (evaluated from the right), arr[0] (within the index of the 2nd element on the left)should still be "4" arr[0] -1 (the index of the 2nd element on the left) should thus be "3"

Answer: Because the target list does not get evaluated simultaneously. Here is the relevant section of the docs:  The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets.  Two things to keep in mind, the right hand side evaluates the expression first. So on the RHS, we first create the tuple :  (3, 4)  Note, that is done left to right. Now, the assignment to each target in the target list on the left is done in order: arr[0] = 3  Then the next target, arr[0] is 3, and 3-1 is 2 arr[2] = 4  So a simple solution is to just to compute the indices first before the swap: >>> arr = [4, 2, 1, 3] >>> i, j = arr[0] - 1, 0 >>> arr[j], arr[i] = arr[i], arr[j] >>> arr [3, 2, 1, 4]  Here is a demonstration using a verbose list that we can define easily: >>> class NoisyList(list): ... def __getitem__(self, item): ... value = super().__getitem__(item) ... print("GETTING", item, "value of", value) ... return value ... def __setitem__(self, item, value): ... print("SETTING", item, 'with', value) ... super().__setitem__(item, value) ... >>> arr = NoisyList([4, 2, 1, 3]) >>> arr[0], arr[arr[0]-1] = arr[arr[0]-1], arr[0] GETTING 0 value of 4 GETTING 3 value of 3 GETTING 0 value of 4 SETTING 0 with 3 GETTING 0 value of 3 SETTING 2 with 4
Question: Python formatting guidelines, the famous PEP8 recommends no line longer than 79 chars. I can easily auto-format my code to a max line length with the Black Formatter, but it does not break long strings. The linter will still complain about a long URL in your code and Black won't help. Is it possible to automatically break long strings with Black formatter?

Answer: Edit 2024-05-27: updating for new black and VSCode configurations. First you must install VSCode Black Extension. Yes it is possible due to a new feature. First make sure that you have a very recent Black formatter installed. Now just run black with the option --preview. In VSCode you can configure it in your settings.json file: "black-formatter.args": [ "--line-length", "100", "--preview" ],  After you edit settings.json, restart Black server for the change to take effect: Cmd/Ctrl + Shift + P -> Black Formatter: Restart Server. BTW, if you want to increase the default line length, it is a good idea to also change to the same value in your linter:  "flake8.args": [ "--max-line-length=100", ],  Some teams really prefer longer lines, don't let them use this as a reason for not automatically formatting. BTW, PEP8 support greater line length:  Some teams strongly prefer a longer line length. For code maintained exclusively or primarily by a team that can reach agreement on this issue, it is okay to increase the line length limit up to 99 characters, provided that comments and docstrings are still wrapped at 72 characters.
Question: I am just learning to use PyTorch as a beginner. If anyone is familiar with PyTorch, would you tell me the difference between nn.Module and nn.Sequential? My questions are  What is the advantage to use nn.Module instead of nn.Sequential?  Which is regularly utilised to build the model?  How we should select nn.Module or nn.Sequential?  

Answer: TLDR; answering your questions   What is the advantage to use nn.Module instead of nn.Sequential?   While nn.Module is the base class to implement PyTorch models, nn.Sequential is a quick way to define a sequential neural network structures inside or outside an existing nn.Module.   Which is regularly utilized to build the model?   Both are widely used.   How we should select nn.Module or nn.Sequential?   All neural networks are implemented with nn.Module. If the layers are sequentially used (self.layer3(self.layer2(self.layer1(x))), you can leverage nn.Sequential to not have to define the forward function of the model.  I should start by mentioning that nn.Module is the base class for all neural network modules in PyTorch. As such nn.Sequential is actually a direct subclass of nn.Module, you can look for yourself on this line. When creating a new neural network, you would usually go about creating a new class and inheriting from nn.Module, and defining two methods: __init__ (the initializer, where you define your layers) and forward (the inference code of your module, where you use your layers). That's all you need, since PyTorch will handle backward pass with Autograd. Here is an example of a module: class NN(nn.Module): def __init__(self): super().__init__() self.fc1 = nn.Linear(10, 4) self.fc2 = nn.Linear(4, 2) def forward(self, x) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) return x  If the model you are defining is sequential, i.e. the layers are called sequentially on the input, one by one. Then, you can simply use a nn.Sequential. As I explained earlier, nn.Sequential is a special kind of nn.Module made for this particular widespread type of neural network. The equivalent here is: class NN(nn.Sequential): def __init__(self): super().__init__( nn.Linear(10, 4), nn.ReLU(), nn.Linear(4, 2), nn.ReLU())  Or a simpler way of putting it is: NN = Sequential( nn.Linear(10, 4), nn.ReLU(), nn.Linear(4, 2), nn.ReLU())  The objective of nn.Sequential is to quickly implement sequential modules such that you are not required to write the forward definition, it being implicitly known because the layers are sequentially called on the outputs. In a more complicated module though, you might need to use multiple sequential submodules. For instance, take a CNN classifier, you could define a nn.Sequential for the CNN part, then define another nn.Sequential for the fully connected classifier section of the model.
Question: Django has a decorator called cached_property which can be imported from django.utils.functional. On the other hand, Python 3.8 added cached_property to the standard library which can be imported from functools. Are both equivalent, i.e., are they interchangeable? or what is the difference between both? Are there any best practices when to use one or the other?

Answer: After some research both basically work the same way and the only difference you would see would be in the error handling and performance. There is a ticket #30949 on Django's issue tracker to use functools.cached_property instead of django.utils.functional.cached_property. You can see the source code [GitHub] for functools.cached_property and also for django's version [GitHub]. The basic difference is functool's version does a little bit more error handling and the main difference is that functool (pre Python 3.12) uses locking mechanism for thread safety, this causes a performance dip compared to Django's version. From some benchmarking done in the ticket linked above it seems Django's version is much more efficient in terms of performance:  % python benchmark.py ..................... Django Cache: Mean +- std dev: 12.8 ms +- 0.2 ms ..................... Python Cache: Mean +- std dev: 113 ms +- 2 ms   There is also an issue 43468 on Python's bug tracker regarding this. Note that this locking behavior is removed in Python version 3.12 onwards and the performance should be mostly similar now. In summary if you are using Python 3.12+ prefer functools version otherwise use Django's version if thread safety is not an issue otherwise you might want to use the functools version.
Question: The installation on the m1 chip for the following packages: Numpy 1.21.1, pandas 1.3.0, torch 1.9.0 and a few other ones works fine for me. They also seem to work properly while testing them. However when I try to install scipy or scikit-learn via pip this error appears: ERROR: Failed building wheel for numpy Failed to build numpy ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly Why should Numpy be build again when I have the latest version from pip already installed? Every previous installation was done using python3.9 -m pip install ... on Mac OS 11.3.1 with the apple m1 chip. Maybe somebody knows how to deal with this error or if its just a matter of time.

Answer: UPDATE: scikit-learn now works via pip ✅ Just first brew install openblas - it has instructions for different processors (wikipedia) brew install openblas export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas) export CFLAGS="-falign-functions=8 ${CFLAGS}" # ^ no need to add to .zshrc, just doing this once. pip install scikit-learn  Worked great on Apple Silicon M1 🎉 Extra details about how Pip works Pip downloaded the source from Pipy, then built the wheel targeting MacOS X 12.0, and arm64 (apple silicon): scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl. Building wheels for collected packages: scikit-learn Building wheel for scikit-learn (pyproject.toml) ... done Created wheel for scikit-learn: filename=scikit_learn-1.0.1-cp38-cp38-macosx_12_0_arm64.whl size=6364030 sha256=0b0cc9a21af775e0c8077ee71698ff62da05ab62efc914c5c15cd4bf97867b31 Successfully built scikit-learn Installing collected packages: scipy, scikit-learn Successfully installed scikit-learn-1.0.1 scipy-1.7.3  Note on Pipy: we usually download either a pre-built wheel (yay, this is excellent for reliable distribution and ensuring compatability). Or, if no prebuilt wheel exists (sad) then we download a tar.gz and build it ourselves. This happens because the authors don't publish a prebuilt wheel to Pipy, but more and more people are adding this to their CI (github actions) workflow. Building the wheel ourselves takes more cpu time, and is generally less reliable but works in this case. Here we are downloading a pre-built wheel that has very few limitations: it works for any version of python 3, for any os, for any architecture (like amd64 or arm64): click-8.0.3-py3-none-any.whl Collecting click>=7.0 Downloading click-8.0.3-py3-none-any.whl  Here apparently we had no wheel available, so we have to build it ourselves with setuptools running setup.py. Collecting grpcio>=1.28.1 Downloading grpcio-1.42.0.tar.gz (21.3 MB) |████████████████████████████████| 21.3 MB 12.7 MB/s Preparing metadata (setup.py) ... done ## later in the process it installs using setuptools Running setup.py install for grpcio ... done  Good luck and happy piping.
Question: I want to stop a Google Colab notebook programmatically when the thing I want to do has ended, I thought of putting a line at the end that would stop it from running. I have tried these, but none work. They all restart instead of shutting down or then just give an error. I got these from here:  Is there a function in google.colab module to close the runtime https://newbedev.com/google-colab-how-to-restart-runtime-using-python-code-or-command-line-interface  import os os.kill(os.getpid(), 9)  import sys sys.exit()  exit()  quit()  !kill -9 -1  There is a shortcut for ending all runtimes, (It's undefined by default by I added it as Ctrl + Shift + K), so maybe I could write a program that would virtually type those keys. But it opens a popup before shutting down and I may not be there to confirm it. How can I do this? Thanks!

Answer: Now (since September 12th, 2022) you can do this: from google.colab import runtime runtime.unassign()  It was announced in this GitHub issue: https://github.com/googlecolab/colabtools/issues/2568. Thanks @HappyFace for commenting this link.
Question: Hi I want to ask if it's possible to enable auto closing curly brackets in f-strings in Visual Studio Code. In Python you use often f-strings and therefore you need curly brackets. print(f"Hello {name}!")  I already found something but I don't know if that feature is already implemented and if not if I can implement it with external plugins?: https://github.com/microsoft/vscode-python/issues/13673

Answer: Use the "Always" Auto Closing Brackets setting, but limit it to Python in your settings.json: "[python]": { "editor.autoClosingBrackets": "always" }
Question: In Spark, literal columns, when added, are not nullable: from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.getOrCreate() df = spark.createDataFrame([(1,)], ['c1']) df = df.withColumn('c2', F.lit('a')) df.printSchema() # root # |-- c1: long (nullable = true) # |-- c2: string (nullable = false)  How to create a nullable column?

Answer: The shortest method I've found - using when (the otherwise clause seems not needed): df = df.withColumn('c2', F.when(F.lit(True), F.lit('a')))  If in Scala: .withColumn("c2", when(lit(true), lit("a")))  Full test result: from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.getOrCreate() df = spark.createDataFrame([(1,)], ['c1']) df = df.withColumn('c2', F.when(F.lit(True), F.lit('a'))) df.show() # +---+---+ # | c1| c2| # +---+---+ # | 1| a| # +---+---+ df.printSchema() # root # |-- c1: long (nullable = true) # |-- c2: string (nullable = true)