eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a FastAPI app with a route prefix as /api/v1. When I run the test it throws 404. I see this is because the TestClient is not able to find the route at /ping, and works perfectly when the route in the test case is changed to /api/v1/ping. Is there a way in which I can avoid changing all the routes in all the test functions as per the prefix? This seems to be cumbersome as there are many test cases, and also because I dont want to have a hard-coded dependency of the route prefix in my test cases. Is there a way in which I can configure the prefix in the TestClient just as we did in app, and simply mention the route just as mentioned in the routes.py? routes.py from fastapi import APIRouter router = APIRouter() @router.get("/ping") async def ping_check(): return {"msg": "pong"}  main.py from fastapi import FastAPI from routes import router app = FastAPI() app.include_router(prefix="/api/v1")  In the test file I have: test.py from main import app from fastapi.testclient import TestClient client = TestClient(app) def test_ping(): response = client.get("/ping") assert response.status_code == 200 assert response.json() == {"msg": "pong"} 

Answer: Figured out a workaround for this. The TestClient has an option to accept a base_url, which is then urljoined with the route path. So I appended the route prefix to this base_url.  source: url = urljoin(self.base_url, url)  However, there is a catch to this - urljoin concatenates as expected only when the base_url ends with a / and the path does not start with a /. This SO answer explains it well. This resulted in the below change: test.py from main import app, ROUTE_PREFIX from fastapi.testclient import TestClient client = TestClient(app) client.base_url += ROUTE_PREFIX # adding prefix client.base_url = client.base_url.rstrip("/") + "/" # making sure we have 1 and only 1 `/` def test_ping(): response = client.get("ping") # notice the path no more begins with a `/` assert response.status_code == 200 assert response.json() == {"msg": "pong"}
Question: I am experiencing some problems using the TextBlob library. I'm trying to run a very simple piece of code like this: from textblob import TextBlob text = 'this is just a test' blob = TextBlob(text) blob.detect_language()  And it continually gives me this error: /usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler): HTTPError: HTTP Error 404: Not Found  What is the problem? I have tried it on several devices and it gives me the same error every time.

Answer: Function dectect_language() sends a request to google translate service: http://translate.google.com/translate_a/t?client=webapp&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t&dt=at&ie=UTF-8&oe=UTF-8&otf=2&ssel=0&tsel=0&kc=1&sl=auto&tk=276174.132528 and this url returns 404. From documentation on detect_language()  Deprecated since version 0.16.0: Use the official Google Translate API instead.  I wouldn't count on this function to be working in the future.
Question: I am trying to find a very nice python idiom to use aws boto3 paginators in the most "pythonic" way. Below is the best I have been able to come up with and I'm still not happy with it. Any ideas on how to make pagination simpler, possibly not using while True:? import boto3 client = boto3.client('acm', region_name='ap-southeast-2') paginator = client.get_paginator('list_certificates') response_iterator = paginator.paginate() while True: for certificates in response_iterator: for certificate in certificates['CertificateSummaryList']: print(certificate) if response_iterator.resume_token: response_iterator = paginator.paginate( PaginationConfig={ 'StartingToken': response_iterator.resume_token }) else: break 

Answer: Woudn't the following form work?: client = boto3.client('acm', region_name='ap-southeast-2') paginator = client.get_paginator('list_certificates') for page in paginator.paginate(): print(page)
Question: I have the following expression: >>> a = 3 >>> b = 2 >>> a == (a := b) False  Now, a == 2 after the operation, as expected. And the result is what I would want, i.e., comparison of a to RHS of assignment before assignment. Reversing the order of the equality operator reverses the result: >>> a = 3 >>> b = 2 >>> (a := b) == a True  There does not appear to be anything directly relevant to this corner case in PEP-572, relative precedence section. The next section, change to evaluation order mentions that the evaluation order is left-to-right. Is that what is going on here (stash the value of a, update it, and compare vs update a, then compare against its new value)? Where is this behavior defined, and how reliable is it?

Answer: Neither of those PEP sections have to do with this. You just have a == comparison, and the general Evaluation order applies: "Python evaluates expressions from left to right." So your (a := b) == a simply first evaluates the left side (a := b), assigning something to a and evaluating to the same value. And then evaluate the right side a, which is of course still the same (just-assigned) value, so you get True.  About those PEP sections: What that first PEP section says is that := groups less tightly than ==, so it would apply if you didn't have parentheses:  a == a := b would mean (a == a) := b (you'd get a syntax error for trying to assign to a comparison). a := b == a would mean a := (b == a), where with your values the b == a evaluates to False and that gets assigned to a and becomes the result of the entire expression. (Note that at statement-level, you'd have to write (a := b == a).)  What that second PEP section does is just to point out something bad that had already existed but which the := made "more visible", so they suggested to finally fix it. The issue was that a dict comprehension like {X: Y for ...} evaluated Y before X, against the general left-to-right rule and against dict displays like {X: Y} which already did evaluate X before Y as expected. Consider this: >>> a, b = 3, 2 >>> {a: (a := b) for _ in '_'} {3: 2}  With that old behavior, it would've resulted in {2: 2}. And given that people might write something like that when := became available, it became more of an issue.
Question: I have an anaconda virtual environment that I wish to use. I am able to use Select Interpreter, which finds and allows me to accurately select said virtual environment. I am also able to use this with jupyter notebooks. What I am not able to do is have the integrated terminal automatically activate this environment. Every time I open a new terminal, I have: (base) PS C:\working_folder  If I manually activate the environment in the integrated terminal, I am then able to use it. My issue is that I don't want to have to remember to manually activate it. Things I've tried:  Setting "Python.terminal.activateEnvironment" to true Setting "Python.terminal.activateEnvInCurrentTerminal" to true Updating pythonPath in my working folder settings.json 

Answer: Open Powershell and run as Administrator, execute the following command conda config --set auto_activate_base true  Then restart VS Code, add the following in User Settings.json: "python.defaultInterpreterPath": "\\path\\to\\conda\\python.exe", "Python.terminal.activateEnvironment": true, "Python.terminal.activateEnvInCurrentTerminal": true  Reload Window from Command Palette, select base:conda as python interpreter then press Ctrl+Shift+` to open a new integrated Terminal, conda environment should be activated automatically in it.
Question: I have used python's sklearn package for K-means clustering. So far I am able to get the coordinates of the cluster centers using the following code. import numpy as np from sklearn.cluster import KMeans p50 = np.load('tsnep400.npy') kmeans = KMeans(n_clusters=50).fit(p50) np.savetxt('kmeans_50clusters_centers_tsnep400', kmeans.cluster_centers_, fmt='%1.3f') np.savetxt('kmeans_50clusters_tsnep400.dat', kmeans.labels_, fmt='%1.1d') centroids = {i: np.where(kmeans.labels_ == i)[0] for i in range(kmeans.n_clusters)} np.save('kmeans_50clusters_memebers_tsnep400.npy',centroids)  How do I find the index of the point closest to cluster centers?

Answer: According to the scikit-learn documentation, the attribute .labels_ contains the labels of each point, by their index. Thus, you can use this to group each of your points into a cluster and then calculate the distance to each cluster center. You can use the following code for this: from scipy.spatial.distance import euclidean # Loop over all clusters and find index of closest point to the cluster center and append to closest_pt_idx list. closest_pt_idx = [] for iclust in range(kmeans.n_clusters): # get all points assigned to each cluster: cluster_pts = p50[kmeans.labels_ == iclust] # get all indices of points assigned to this cluster: cluster_pts_indices = np.where(kmeans.labels_ == iclust)[0] cluster_cen = kmeans.cluster_centers_[iclust] min_idx = np.argmin([euclidean(p50[idx], cluster_cen) for idx in cluster_pts_indices]) # Testing: print('closest point to cluster center: ', cluster_pts[min_idx]) print('closest index of point to cluster center: ', cluster_pts_indices[min_idx]) print(' ', p50[cluster_pts_indices[min_idx]]) closest_pt_idx.append(cluster_pts_indices[min_idx])
Question: pip3 install PyMuPDF Collecting PyMuPDF Using cached PyMuPDF-1.18.17-cp37-cp37m-win_amd64.whl (5.4 MB) Installing collected packages: PyMuPDF Successfully installed PyMuPDF-1.18.17  import fitz doc = fitz.open("my_pdf.pdf")  When I look for def open on the fitz.py file, I find nothing. So I understand the error But I don't understand why the file that I download doesn't have this function ? Can someone share the good files please ? Or maybe I missed something else ? FULL TRACE: runfile('D:/Documents/Python_projects/Point_and_area_pdf_to_excel/get_info.py', wdir='D:/Documents/Python_projects/Point_and_area_pdf_to_excel') Reloaded modules: six, dateutil._common, dateutil.relativedelta, dateutil.tz._common, dateutil.tz._factories, dateutil.tz.win, dateutil.tz.tz, dateutil.tz, dateutil.parser._parser, dateutil.parser.isoparser, dateutil.parser, chardet.enums, chardet.charsetprober, chardet.charsetgroupprober, chardet.codingstatemachine, chardet.escsm, chardet.escprober, chardet.latin1prober, chardet.mbcssm, chardet.utf8prober, chardet.mbcharsetprober, chardet.euctwfreq, chardet.euckrfreq, chardet.gb2312freq, chardet.big5freq, chardet.jisfreq, chardet.chardistribution, chardet.jpcntx, chardet.sjisprober, chardet.eucjpprober, chardet.gb2312prober, chardet.euckrprober, chardet.cp949prober, chardet.big5prober, chardet.euctwprober, chardet.mbcsgroupprober, chardet.hebrewprober, chardet.sbcharsetprober, chardet.langbulgarianmodel, chardet.langgreekmodel, chardet.langhebrewmodel, chardet.langrussianmodel, chardet.langthaimodel, chardet.langturkishmodel, chardet.sbcsgroupprober, chardet.universaldetector, chardet.version, chardet Traceback (most recent call last): File "D:\Documents\Python_projects\Point_and_area_pdf_to_excel\get_info.py", line 45, in <module> print(get_dict_list(path)) File "D:\Documents\Python_projects\Point_and_area_pdf_to_excel\get_info.py", line 7, in get_dict_list text_list = get_pdf_page_text_list(pdf_path) File "D:\Documents\Python_projects\Point_and_area_pdf_to_excel\get_info.py", line 19, in get_pdf_page_text_list doc = fitz.open(pdf_path) AttributeError: module 'fitz' has no attribute 'open' 

Answer: This is likely to be an installation issue and looks like there already exists a package fitz installed on your environment and is unrelated to PyMuPDF. So when PyMuPDF calls fitz it might actually be calling the wrong fitz package. You can consider doing a clean install of all dependencies or create a virtual environment to work with PyMuPDF. You can also try rolling back fitz to version 1.16.14
Question: If I am running a container within Cloud Run and do a print statement in my python code. Where can I view it? Cloud logs seem to show logs for the contain itself(build, etc)? to debug my code often I do write statements that help me figure what's going on. Where would that print output be located?

Answer: 1] You can find all the logs including your print statement output in Cloud Logging as mentioned in this link. So when you write a print statement from your service they will be automatically picked up by Cloud Logging. 2] Steps to view logs in Cloud Logging: Logs Explorer -> Cloud Run Revision. 3] You may wanna check your logging level. For example: if you have configured level as logging.ERROR in basicConfig (default is WARNING), and used logging.info() in your code, then it will not be printed. You can refer to this link for more information. 4] Also, you may try flushing the stdout which will make sure the logs get written from buffers. You may refer Stackoverflow answer on how to do this.
Question: I have a project my-project that uses a submodule my-submodule. The submodule has dependencies different from my-project in poetry.lock & pyproject.toml files. I have installed the dependencies required for my-project using poetry add. These deps are installed and poetry.lock & pyproject.toml files are created in the root folder of my-project. Now, I would also like to install the dependencies of the submodule. Assuming that the path of the submodule is path/to/submodule/from/root, how can I install the dependencies of the submodule and make those deps reflect in the poetry.lock & pyproject.toml files of the root? A similar question has been asked here: Manage dependencies of git submodules with poetry, but there isn't a solution provided there.

Answer: You can declare the submodule as a path dependency in the pyproject.toml of the parent project. It will then treat the submodule as a package and include it in dependency install/resolution. Be sure to also include the develop attribute when declaring the dependency, as follows: [tool.poetry.dependencies] my-package = { path = "./path/to/submodule/from/root", develop = true }  Link to docs: https://python-poetry.org/docs/dependency-specification/#path-dependencies
Question: When I am trying to reset the index of my dataFrame, it is not working. new = pd.DataFrame(columns=['a','b','Amount1']) new['Amount1'] = [0,1,6,7,8,9] new['a'] = ['sarim',1,2,3,4,'sarim'] df_tf = new[new['a']=='sarim']['Amount1'] df_tf.reset_index(inplace=True) ret_df['Amount1'] = df_tf 

Answer: you can try, df_tf.reset_index(drop = True, inplace=True) ret_df['Amount1'] = df_tf  or ret_df['Amount1'] = list(df_tf)
Question: I am trying to create a telegram bot. The code i am trying to execute is : from telegram import ParseMode  But it is throwing up this error: ImportError: cannot import name 'ParseMode' from 'telegram' (C:\ProgramData\Anaconda3\lib\site-packages\telegram\__init__.py)  Could you please advise how to fix this error?

Answer: you have to import with this way: from telegram.ext import ParseMode   if problem not solved: install the package like this:  pip install python_telegram_bot  or  pip install "python_telegram_bot==12.4.2"
Question: Currently for my scientific experiments I use dbg = print # def dbg(*args): pass  So I have a lot of dbg(x, y, f(x)) in code, all of which I can "turn off" my commenting one line and uncommenting another. However, the output looks brief, e.g. 0 15 32. Is there a way to make it look like x = 0, y = 15, f(x) = 32? I tried to write something using eval, but couldn't.

Answer: Try using the = operator on f-strings: dbg(f"{x=}, {y=}, {f(x)=}")  This was introduced in Python3.8 f-strings support = for self-documenting expressions and debugging  Added an = specifier to f-strings. An f-string such as f'{expr=}' will expand to the text of the expression, an equal sign, then the representation of the evaluated expression. For example: >>> user = 'eric_idle' >>> member_since = date(1975, 7, 31) >>> f'{user=} {member_since=}' "user='eric_idle' member_since=datetime.date(1975, 7, 31)"
Question: I have a pandas dataframe of pairwise distances in the form of:  SampleA SampleB Num_Differences 0 sample_1 sample_2 1 1 sample_1 sample_3 4 2 sample_2 sample_3 8  Note that there are no self-self comparisons (e.g., sample_1 vs sample_1 won't be represented). I would like to convert this table into a squareform distance matrix instead, like so:  sample_1 sample_2 sample_3 sample_1 1 4 sample_2 1 8 sample_3 4 8  Can anyone give me some pointers on how to do such a conversion in python? The problem is analogous to a previous question in R (Converting pairwise distances into a distance matrix in R), but I don't know the corresponding python functions to use. The problem also appears to be the opposite of this question (Convert a distance matrix to a list of pairwise distances in Python). Some code to reproduce a dataframe in the form I'm using: df = pd.DataFrame([['sample_1', 'sample_2', 1], ['sample_1', 'sample_3', 4], ['sample_2', 'sample_3', 8]], columns=['SampleA', 'SampleB', 'Num_Differences']) 

Answer: You can reshape to square, and then make symmetrical by adding the transposed values: # make unique, sorted, common index idx = sorted(set(df['SampleA']).union(df['SampleB'])) # reshape (df.pivot(index='SampleA', columns='SampleB', values='Num_Differences') .reindex(index=idx, columns=idx) .fillna(0, downcast='infer') .pipe(lambda x: x+x.values.T) )  Alternatively, you can use ordered categorical indexes and keep NAs during reshaping with pivot_table. Then add the transposed values to make symmetrical: cat = sorted(set(df['SampleA']).union(df['SampleB'])) (df.assign(SampleA=pd.Categorical(df['SampleA'], categories=cat, ordered=True), SampleB=pd.Categorical(df['SampleB'], categories=cat, ordered=True), ) .pivot_table(index='SampleA', columns='SampleB', values='Num_Differences', dropna=False, fill_value=0) .pipe(lambda x: x+x.values.T) )  Output: SampleB sample_1 sample_2 sample_3 SampleA sample_1 0 1 4 sample_2 1 0 8 sample_3 4 8 0
Question: After playing with some "defect" scenarios with celery (Redis being a broker for whatever it worth) we came to understanding that there is effectively no sense in setting acks_late=true without simultaneous setting of task_reject_on_worker_lost=true because the task won't be rescheduled (again, in our tests) -- task stays in the "unacked" category forever. At the same time everybody says that acks_late will make the task being subject for rescheduling on the same / another worker, so the question is: when does it happen? The official docs say that  Note that the worker will acknowledge the message if the child process executing the task is terminated (either by the task calling sys.exit(), or by signal) even when acks_late is enabled. This behavior is intentional as…  We don’t want to rerun tasks that forces the kernel to send a SIGSEGV (segmentation fault) or similar signals to the process.  We assume that a system administrator deliberately killing the task does not want it to automatically restart.  A task that allocates too much memory is in danger of triggering the kernel OOM killer, the same may happen again.  A task that always fails when redelivered may cause a high-frequency message loop taking down the system.   If you really want a task to be redelivered in these scenarios you should consider enabling the task_reject_on_worker_lost setting.  What are possible examples of "something went wrong" that don't fall into the "worker terminated deliberately or due to a signal caught" category?

Answer: Reboot, power outage, hardware failure. n.b., all of your examples assume that the prefetch multiplier is 1.
Question: I'm trying to set different colors for some bars in a plotly express bar graph: import plotly.express as px import pandas as pd data = {'Name':['2020/01', '2020/02', '2020/03', '2020/04', '2020/05', '2020/07', '2020/08'], 'Value':[34,56,66,78,99,55,22]} df = pd.DataFrame(data) color_discrete_sequence = ['#ec7c34']*len(df) color_discrete_sequence[5] = '#609cd4' fig=px.bar(df,x='Name',y='Value',color_discrete_sequence=color_discrete_sequence) fig.show()  My expectations were that one (the sixth one) bar had a different color, however I got this result:  What am I doing wrong?

Answer: This happens because color in px.bar is used to name a category to illustrate traits or dimensions of a dataset using a colorscale. Or in you your case, rather a color cycle since you're dealing with a categorical / discrete case. color_discrete_sequence is then used to specify which color sequence to follow. One way to achieve your goal using your setup here, is to simply define a string variable with unique values, for example like df['category'] [str(i) for i in df.index], and then use: fig=px.bar(df,x='Name',y='Value', color = 'category', color_discrete_sequence=color_discrete_sequence, )  Plot:  If df['category'] is a numerical value, color_discrete_sequence will be ignored, and a default continuous sequence will be applied:  If anything else is unclear, don't hesitate to let me know. Complete code: import plotly.express as px import pandas as pd data = {'Name':['2020/01', '2020/02', '2020/03', '2020/04', '2020/05', '2020/07', '2020/08'], 'Value':[34,56,66,78,99,55,22]} df = pd.DataFrame(data) df['category'] = [str(i) for i in df.index] # df['category'] = df.index color_discrete_sequence = ['#ec7c34']*len(df) color_discrete_sequence[5] = '#609cd4' fig=px.bar(df,x='Name',y='Value', color = 'category', color_discrete_sequence=color_discrete_sequence, ) fig.show()
Question: I am reading JSON file that has some date columns. The issue is some of the date columns contain dates in Arabic/urdu text : ٠٤-٢٥-٢٠٢١  I want to convert it to English date in yyyy-mm-dd format. How to achieve this in Pyspark?

Answer: You can convert arabic number to english by casting type to decimal. df = spark.createDataFrame([('٠٤-٢٥-٢٠٢١',)],['arabic']) df.withColumn('split', split('arabic', '-')) \ .withColumn('date', concat_ws('-', col('split')[2].cast('decimal'), col('split')[0].cast('decimal'), col('split')[1].cast('decimal'))) \ .drop('split').show() +----------+---------+ | arabic| date| +----------+---------+ |٠٤-٢٥-٢٠٢١ |2021-4-25| +----------+---------+
Question: I am puzzled by the behavior I observe in the following example: import tensorflow as tf @tf.function def f(a): c = a * 2 b = tf.reduce_sum(c ** 2 + 2 * c) return b, c def fplain(a): c = a * 2 b = tf.reduce_sum(c ** 2 + 2 * c) return b, c a = tf.Variable([[0., 1.], [1., 0.]]) with tf.GradientTape() as tape: b, c = f(a) print('tf.function gradient: ', tape.gradient([b], [c])) # outputs: tf.function gradient: [None] with tf.GradientTape() as tape: b, c = fplain(a) print('plain gradient: ', tape.gradient([b], [c])) # outputs: plain gradient: [<tf.Tensor: shape=(2, 2), dtype=float32, numpy= # array([[2., 6.], # [6., 2.]], dtype=float32)>]  The lower behavior is what I would expect. How can I understand the @tf.function case? Thank you very much in advance! (Note that this problem is distinct from: Missing gradient when using tf.function , since here all calculations are inside the function.)

Answer: Gradient tape does not record the operations inside the tf.Graph generated by @tf.function treating the function as a whole. Roughly, f is applied to a, and gradient tape has recorded the gradients of the outputs of f with respect to input a (it is the only watched variable, tape.watched_variables()). In the second case, there is no graph generated, and operations are applied in Eager mode. So everything works as expected. A good practice is to wrap a most computationally expensive function in the @tf.function (often a training loop). In your case, it will be smth like: @tf.function def f(a): with tf.GradientTape() as tape: c = a * 2 b = tf.reduce_sum(c ** 2 + 2 * c) grads = tape.gradient([b], [c]) print('tf.function gradient: ', grads) return grads