eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Getting this error Trying to connect elasticsearch docker container with elasticsearch-python client.  /home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elasticsearch/_sync/client/__init__.py:379: SecurityWarning: Connecting to 'https://localhost:9200' using TLS with verify_certs=False is insecure **transport_kwargs, <Elasticsearch(['https://localhost:9200'])> Traceback (most recent call last): File "test_all.py", line 29, in <module> resp = es.index(index="test-index", id=1, document=doc) File "/home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elasticsearch/_sync/client/utils.py", line 404, in wrapped return api(*args, **kwargs) File "/home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elasticsearch/_sync/client/__init__.py", line 2218, in index __method, __path, params=__query, headers=__headers, body=__body File "/home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elasticsearch/_sync/client/_base.py", line 295, in perform_request client_meta=self._client_meta, File "/home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elastic_transport/_transport.py", line 334, in perform_request request_timeout=request_timeout, File "/home/raihan/dev/aims_lab/ai_receptionist/env/lib/python3.6/site-packages/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request raise err from None elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852)))  contents in elastic.py  host = "https://localhost:9200" es = Elasticsearch(host, ca_certs=False, verify_certs=False) print(es) doc = { 'author': 'kimchy', 'text': 'Elasticsearch: cool. bonsai cool.', 'timestamp': datetime.now(), } resp = es.index(index="test-index", id=1, document=doc) print(resp['result']) resp = es.get(index="test-index", id=1) print(resp['_source'])  contents in elasticsearch dockerfile FROM docker.elastic.co/elasticsearch/elasticsearch:7.12.0 RUN elasticsearch-plugin install --batch https://github.com/alexklibisz/elastiknn/releases/download/7.12.0.0/elastiknn-7.12.0.0.zip  urllib3==1.26.9 requests==2.27.1

Answer: #disable certificate es = Elasticsearch(hosts="https://localhost:9200", basic_auth=(USER, PASS), verify_certs=False) #if getting an issue relevant to the certificate then: es = Elasticsearch(hosts="https://localhost:9200", basic_auth=(USER, PASS), ca_certs=CERTIFICATE, verify_certs=False) # I hope you know where to find certificate, e.g: $ find /usr/share/elasticsearch -name "certs.pem"  Reference Follow for more
Question: I am always getting the same error regarding TensorFlow: ModuleNotFoundError: No module named 'tensorflow.contrib'. I am actually using Python version 3.9 but, reading online, it seems that version 3.7 is the last stable one that can work with TensorFlow version >2.0. Unfortunately I have started my project in a venv with the wrong version of Python and I would like to downgrade it, how can I do that?

Answer: Building on @chepner's comment above, since venvs are just directories, you can save your current state and start a fresh virtual environment instead. # Save current installs (venv) -> pip freeze -r > requirements.txt # Shutdown current env (venv) -> deactivate # Copy it to keep a backup -> mv venv venv-3.9 # Ensure you have python3.7 -> python3.7 -V # Create and activate a 3.7 venv -> python3.7 -m venv venv-3.7 -> source venv-3.7/bin/activate # Reinstall previous requirements (venv-3.7) -> pip install -r requirements.txt # Install new requirements  Hope that helps!
Question: I am making a rick roll site for Discord and I would like to redirect to the rick roll page on 404 response status codes. I've tried the following, but didn't work:  @app.exception_handler(fastapi.HTTPException) async def http_exception_handler(request, exc): ... 

Answer: Update A more elegant solution would be to use a custom exception handler, passing the status code of the exception you would like to handle, as shown below: from fastapi.responses import RedirectResponse from fastapi.exceptions import HTTPException @app.exception_handler(404) async def not_found_exception_handler(request: Request, exc: HTTPException): return RedirectResponse('https://fastapi.tiangolo.com')  or, use the exception_handlers parameter of the FastAPI class like this: async def not_found_error(request: Request, exc: HTTPException): return RedirectResponse('https://fastapi.tiangolo.com') exception_handlers = {404: not_found_error} app = FastAPI(exception_handlers=exception_handlers)  Note: In the examples above, a RedirectResponse is returned, as OP asked for redirecting the user. However, you could instead return some custom Response, HTMLResponse or Jinja2 TemplateResponse, as demosntrated in the example below. Working Example app.py from fastapi import FastAPI, Request from fastapi.templating import Jinja2Templates from fastapi.exceptions import HTTPException async def not_found_error(request: Request, exc: HTTPException): return templates.TemplateResponse('404.html', {'request': request}, status_code=404) async def internal_error(request: Request, exc: HTTPException): return templates.TemplateResponse('500.html', {'request': request}, status_code=500) templates = Jinja2Templates(directory='templates') exception_handlers = { 404: not_found_error, 500: internal_error } app = FastAPI(exception_handlers=exception_handlers)  templates/404.html <!DOCTYPE html> <html> <title>Not Found</title> <body> <h1>Not Found</h1> <p>The requested resource was not found on this server.</p> </body> </html>  templates/500.html <!DOCTYPE html> <html> <title>Internal Server Error</title> <body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request. </p> </body> </html>   Original answer You would need to create a middleware and check for the status_code of the response. If it is 404, then return a RedirectResponse. Example: from fastapi import Request from fastapi.responses import RedirectResponse @app.middleware("http") async def redirect_on_not_found(request: Request, call_next): response = await call_next(request) if response.status_code == 404: return RedirectResponse("https://fastapi.tiangolo.com") else: return response
Question: I'm trying to insert bullet point text with docx-template. I know that it can be done with the standard docx like below; document.add_paragraph('text to be bulleted', style='List Bullet')  But I just can't get the same thing to work on docx-template; rt = RichText() rt.add('text to be bulleted', style='List Bullet')  The above code just returns 'text to be bulleted' without the bullet in front. I've looked through all the "tests" on the github and nothing mentions bullet. Any assistance would be greatly appreciated. Thank you in advance.

Answer: Until this is officially supported (open issue), you might use a workaround (but only fixed indentations): In your word document add this (with a real single bullet list item): {% for bullet in bullets %} ● {{ bullet }}{% endfor %}  Note that the {% endfor %} must be in the same line to avoid blank lines between the bullet items. Use this python code here: from docxtpl import DocxTemplate tpl=DocxTemplate('template.docx') context = { 'bullets': [ 'item 1', 'item 2', ], } tpl.render(context) tpl.save("output.docx")
Question: I am trying to delete an element in a json file, here is my json file: before: { "names": [ { "PrevStreak": false, "Streak": 0, "name": "Brody B#3719", "points": 0 }, { "PrevStreak": false, "Streak": 0, "name": "XY_MAGIC#1111", "points": 0 } ] }  after running script:  { "names": [ { "PrevStreak": false, "Streak": 0, "name": "Brody B#3719", "points": 0 } ] }  how would I do this in python? the file is stored locally and I am deciding which element to delete by the name in each element Thanks

Answer: You will have to read the file, convert it to python native data type (e.g. dictionary), then delete the element and save the file. In your case something like this could work: import json filepath = 'data.json' with open(filepath, 'r') as fp: data = json.load(fp) del data['names'][1] with open(filepath, 'w') as fp: json.dump(data, fp)
Question: I am trying to set up sqlfluff but for all of our queries it always returns this when running sqlfluff fix [1 templating/parsing errors found]  Is there any way how I can force it to tell me the error that occurs? I tried running it on highes verbosity level but no useful information logged.

Answer: Use the command sqlfluff parse, which will give you the line numbers of where the parse violations are occurring. Once you've rectified all parse violations, run sqlfluff fix again.
Question: question 1) The answer to this question suggested that for a binary classification problem I could use num_labels as 1 (positive or not) or 2 (positive and negative). Is there any guideline regarding which setting is better? It seems that if we use 1 then probability would be calculated using sigmoid function and if we use 2 then probabilities would be calculated using softmax function. question 2) In both cases are my y labels going to be same? each data point will have 0 or 1 and not one hot encoding? For example, if I have 2 data points then y would be 0,1 and not [0,0],[0,1] I have very unbalanced classification problem where class 1 is present only 2% of times. In my training data I am oversampling question 3) My data is in pandas dataframe and I am converting it to a dataset and creating y variable using below. How should I cast my y column - label if I am planning to use num_labels=1? `train_dataset=Dataset.from_pandas(train_df).cast_column("label", ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None))` 

Answer: Well, it probably is kind of late. But I want to point out one thing, according to the Hugging Face code, if you set num_labels = 1, it will actually trigger the regression modeling, and the loss function will be set to MSELoss(). You can find the code here. Also, in their own tutorial, for a binary classification problem (IMDB, positive vs. negative), they set num_labels = 2. from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)  Here is the link.
Question: I'm using VSCode Test Explorer to run my Python unit tests. There was a bug in my code and my tested method never finishes. How do I interrupt my test? I can't find how to do it using the GUI. I had to close VSCode to interrupt it. I'm using pytest framework.

Answer: Silly me, here is the Stop button at the top right of the the Testing tab:
Question: I have a dataframe and I'm using seaborn pairplot to plot one target column vs the rest of the columns. Code is below, import seaborn as sns import matplotlib.pyplot as plt tgt_var = 'AB' var_lst = ['A','GH','DL','GT','MS'] pp = sns.pairplot(data=df, y_vars=[tgt_var], x_vars=var_lst) pp.fig.set_figheight(6) pp.fig.set_figwidth(20)  The var_lst is not a static list, I just provided an example. What I need is to plot tgt_var on Y axis and each var_lst on x axis. I'm able to do this with above code, but I also want to use log scale on X axis only if the var_lst item is 'GH' or 'MS', for the rest normal scale. Is there any way to achieve this?

Answer: Iterate pp.axes.flat and set xscale="log" if the xlabel matches "GH" or "MS": log_columns = ["GH", "MS"] for ax in pp.axes.flat: if ax.get_xlabel() in log_columns: ax.set(xscale="log")  Full example with the iris dataset where the petal columns are xscale="log": import seaborn as sns df = sns.load_dataset("iris") pp = sns.pairplot(df) log_columns = ["petal_length", "petal_width"] for ax in pp.axes.flat: if ax.get_xlabel() in log_columns: ax.set(xscale="log")
Question: I have a problem. I am using ArangoDB enterprise:3.8.6 via Docker. But unfortunately my query takes longer than 30s. When it fails, the error is arangodb HTTPConnectionPool(host='127.0.0.1', port=8529): Read timed out. (read timeout=60).  My collection is aroung 4GB huge and ~ 1.2 mio - 900k documents inside the collection.  How could I get the complete collection with all documents without any error? Python code (runs locally on my machine) from arango import ArangoClient # Initialize the ArangoDB client. client = ArangoClient() # Connect to database as user. db = client.db(<db>, username=<username>, password=<password>) cursor = db.aql.execute(f'FOR doc IN students RETURN doc', batch_size=10000) result = [doc for doc in cursor] print(result[0]) [OUT] arangodb HTTPConnectionPool(host='127.0.0.1', port=8529): Read timed out. (read timeout=60)  docker-compose.yml for ArangoDB version: '3.7' services: database: container_name: database__arangodb image: arangodb/enterprise:3.8.6 environment: - ARANGO_LICENSE_KEY=<key> - ARANGO_ROOT_PASSWORD=root - ARANGO_CONNECT_TIMEOUT=300 - ARANGO_READ_TIMEOUT=600 ports: - 8529:8529 volumes: - C:/Users/dataset:/var/lib/arangodb3  What I tried cursor = db.aql.execute('FOR doc IN <Collection> RETURN doc', stream=True) while cursor.has_more(): # Fetch until nothing is left on the server. cursor.fetch() while not cursor.empty(): # Pop until nothing is left on the cursor. cursor.pop() [OUT] CursorNextError: [HTTP 404][ERR 1600] cursor not found # A N D cursor = db.aql.execute('FOR doc IN <Collection> RETURN doc', stream=True, ttl=3600) collection = [doc for doc in cursor] [OUT] nothing # Runs, runs and runs for more than 1 1/2 hours  What worked but only for 100 documents # And that worked cursor = db.aql.execute(f'FOR doc IN <Collection> LIMIT 100 RETURN doc', stream=True) collection = [doc for doc in cursor] 

Answer: You can increase the HTTP client's timeout by using a custom HTTP client for Arango. The default is set here to 60 seconds. from arango.http import HTTPClient class MyCustomHTTPClient(HTTPClient): REQUEST_TIMEOUT = 1000 # Set the timeout you want in seconds here # Pass an instance of your custom HTTP client to Arango: client = ArangoClient( http_client=MyCustomHTTPClient() )
Question: I have difficulties finding return types that satisfy mypy. I have two functions. The first one returns a Union type, since the type depends on the parameter given to the function. The second function calls the first one with a default parameter. Thus, the type is not a Union type -- it can be narrowed down to one of the types in the Union. Let me give you an example of a very simple version of my problem: from typing import Union def first(as_string: bool) -> Union[str, int]: if as_string: return "42" else: return 42 def second(as_string: bool) -> str: return first(True)  This causes an error: Incompatible return value type (got "str", expected "Union[str, int]")  How can I prevent mypy from throwing an error while still using type hints? If you want to suggest splitting the first function, please take in mind that this is just a simplification. My (first) function receives a function (sklearn.metrics function) and will return a scalar most of the time. Only when applying a confusion matrix, the type changes. The first function does some preprocessing and than applies the metric. I simply want to have a differently named function for the confusion matrix since I think it is a special case of a metric.

Answer: mypy can't infer the relationship between the types of the parameters and the types of the return values. You have two options:  Use an assert to make sure that the type is correct: def second(as_string) -> str: ret = first(True) assert isinstance(ret, str) return ret   You can use typing.cast to assert to the typechecker without altering runtime behavior (i.e. without introducing the potential for an AssertionError to be raised). from typing import cast def second(as_string) -> str: return cast(str, first(True))
Question: I am building an audio-based deep learning model. As part of the preporcessing I want to augment the audio in my datasets. One augmentation that I want to do is to apply RIR (room impulse response) function. I am working with Python 3.9.5 and TensorFlow 2.8. In Python the standard way to do it is, if the RIR is given as a finite impulse response (FIR) of n taps, is using SciPy lfilter import numpy as np from scipy import signal import soundfile as sf h = np.load("rir.npy") x, fs = sf.read("audio.wav") y = signal.lfilter(h, 1, x)  Running in loop on all the files may take a long time. Doing it with TensorFlow map utility on TensorFlow datasets: # define filter function def h_filt(audio, label): h = np.load("rir.npy") x = audio.numpy() y = signal.lfilter(h, 1, x) return tf.convert_to_tensor(y, dtype=tf.float32), label # apply it via TF map on dataset aug_ds = ds.map(h_filt)  Using tf.numpy_function: tf_h_filt = tf.numpy_function(h_filt, [audio, label], [tf.float32, tf.string]) # apply it via TF map on dataset aug_ds = ds.map(tf_h_filt)  I have two questions:  Is this way correct and fast enough (less than a minute for 50,000 files)? Is there a faster way to do it? E.g. replace the SciPy function with a built-in TensforFlow function. I didn't find the equivalent of lfilter or SciPy's convolve. 

Answer: Here is one way you could do Notice that tensor flow function is designed to receive batches of inputs with multiple channels, and the filter can have multiple input channels and multiple output channels. Let N be the size of the batch I, the number of input channels, F the filter width, L the input width and O  the number of output channels. Using padding='SAME' it maps an input of shape (N, L, I) and a filter of shape (F, I, O) to an output of shape (N, L, O). import numpy as np from scipy import signal import tensorflow as tf # data to compare the two approaches x = np.random.randn(100) h = np.random.randn(11) # h y_lfilt = signal.lfilter(h, 1, x) # Since the denominator of your filter transfer function is 1 # the output of lfiler matches the convolution y_np = np.convolve(h, x) assert np.allclose(y_lfilt, y_np[:len(y_lfilt)]) # now let's do the convolution using tensorflow y_tf = tf.nn.conv1d( # x must be padded with half of the size of h # to use padding 'SAME' np.pad(x, len(h) // 2).reshape(1, -1, 1), # the time axis of h must be flipped h[::-1].reshape(-1, 1, 1), # a 1x1 matrix of filters stride=1, padding='SAME', data_format='NWC') assert np.allclose(y_lfilt, np.squeeze(y_tf)[:len(y_lfilt)])
Question: Does the code below provide correct exception handling. My goal is, to not attempt the file.write() unless the file was successfully opened and ensure the file is closed. I am not concerned with the exact errors of why the file did not open or why the file did not write. Python version 3.6.9. OS Linux. data = "Some data" filename = "test.txt" try: file = open(filename, 'w+') except: print("Error opening file") else: try: file.write(data) except: print("Error writing to file") finally: file.close() 

Answer: You should basically never use a blanket except; always spell out which exception(s) exactly you want to handle. Here is a refactoring using a context handler, so you can avoid the explicit finally: close data = "Some data" filename = "test.txt" try: with open(filename, 'w+') as file: try: file.write(data) except (IOError, OSError): print("Error writing to file") except (FileNotFoundError, PermissionError, OSError): print("Error opening file")  There may be more exceptions you should enumerate; I rattled these off off the top of my head.
Question: I have a print statement that prints a very long big Pandas DataFrame series out, but I need all the information. When printing, Python gives 0 [{This is a long stateme.....}]  1 [{This is a long stateme.....}] and terminates the print statement with dots. I want to see the entire print statement without Python terminating it short, is there a setting to change to let print statements print infinite values?

Answer: Pandas allows options to see a set number of rows, as well as width pd.set_option("display.max_rows", n) pd.set_option("display.expand_frame_repr", True) pd.set_option('display.width', 1000)  To increase the number of characters printed per column, use: pd.set_option("display.max_colwidth", max_characters) By default, a column only prints up to 50 characters, this option allows you to print an max_characters amount of characters. See the docs below: https://pandas.pydata.org/docs/user_guide/options.html
Question: I'm trying to use flake8 only on 3 specific sub directories: features_v2, rules_v2 and indicators_v2. In order to test if my pattern is correct I tried applying it only to features_v2 at first. so I came up with this pattern: exclude = ^(?!.*features_v2).*$ but sadly it doesn't seems to work. Is flake8 does not support lookaround, or it does and I did something wrong? Is there a better way to use flake on 3 different subdirectories?

Answer: flake8's exclude is not a regex but a glob -- it cannot support what you're looking for may I suggest instead to run flake8 on the directories you want? flake8 features_v2 rules_v2 indicators_v2   disclaimer: I maintain flake8
Question: what is an elegant way to access the urls of tweets pictures with tweepy v2? Twitter released a v2 of their API and tweepy adjusted their python module to it (tweepy v2). Lets say for example I have a dataframe of tweets created with tweepy, holding tweet id and so on. There is a row with this example tweet: https://twitter.com/federalreserve/status/1501967052080394240 The picture is saved under a different url and the documentation on tweepy v2 does reveal if there is a way to access it. https://pbs.twimg.com/media/FNgO9vNXIAYy2LK?format=png&name=900x900 Reading thought the requests json obtained through tweepy.Client(bearer_token, return_type = requests.Response) did not hold the required links. I am already using the following parameters in the client: client.get_liked_tweets(user_id, tweet_fields = ['created_at', 'text', 'id', 'attachments', 'author_id', 'entities'], media_fields=['preview_image_url', 'url'], user_fields=['username'], expansions=['attachments.media_keys', 'author_id'] )  What would be a way to obtain or generate the link to the picture of the tweet? Preferably through tweepy v2 itself? Thank you in advance.

Answer: The arguments of get_liked_tweets seem to be correct. Have you looked in the includes dict at the root of the response? The media fields are not in data, so you should have something like that: { "data": { "attachments": { "media_keys": [ "16_1211797899316740096" ] }, "author_id": "2244994945", "id": "1212092628029698048", "text": "[...]" }, "includes": { "media": [ { "media_key": "16_1211797899316740096", "preview_image_url": "[...]", "url": "[...]" } ], "users": [ { "id": "2244994945", "username": "TwitterDev" } ] } }