eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Recently I had to reinstall Python due to a corrupt executable. This made one of our Python scripts bomb with the following error:  AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'  The line of code that caused it to bomb was: from apiclient.discovery import build  I tried pip uninstalling and pip upgrading the google-api-python-client, but I can’t seem to find any information on this particular error. For what it is worth, I am trying to pull Google Analytics information down via an API call. Here is an output of the command prompt error:  File "C:\Analytics\Puritan_GoogleAnalytics\Google_Conversions\mcfTest.py", line 1, in <module> from apiclient.discovery import build File "C:\ProgramData\Anaconda3\lib\site-packages\apiclient\__init__.py", line 3, in <module> from googleapiclient import channel, discovery, errors, http, mimeparse, model File "C:\ProgramData\Anaconda3\lib\site-packages\googleapiclient\discovery.py", line 57, in <module> from googleapiclient import _auth, mimeparse File "C:\ProgramData\Anaconda3\lib\site-packages\googleapiclient\_auth.py", line 34, in <module> import oauth2client.client File "C:\ProgramData\Anaconda3\lib\site-packages\oauth2client\client.py", line 45, in <module> from oauth2client import crypt File "C:\ProgramData\Anaconda3\lib\site-packages\oauth2client\crypt.py", line 45, in <module> from oauth2client import _openssl_crypt File "C:\ProgramData\Anaconda3\lib\site-packages\oauth2client\_openssl_crypt.py", line 16, in <module> from OpenSSL import crypto File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\crypto.py", line 1517, in <module> class X509StoreFlags(object): File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\crypto.py", line 1537, in X509StoreFlags CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK' 

Answer: Upgrade the latest version of PyOpenSSL. python3 -m pip install pip --upgrade pip install pyopenssl --upgrade
Question: I use OpenAI's Whisper python lib for speech recognition. How can I get word-level timestamps?  To transcribe with OpenAI's Whisper (tested on Ubuntu 20.04 x64 LTS with an Nvidia GeForce RTX 3090): conda create -y --name whisperpy39 python==3.9 conda activate whisperpy39 pip install git+https://github.com/openai/whisper.git sudo apt update && sudo apt install ffmpeg whisper recording.wav whisper recording.wav --model large  If using an Nvidia GeForce RTX 3090, add the following after conda activate whisperpy39: pip install -f https://download.pytorch.org/whl/torch_stable.html conda install pytorch==1.10.1 torchvision torchaudio cudatoolkit=11.0 -c pytorch 

Answer: In openai-whisper version 20231117, you can get word level timestamps by setting word_timestamps=True when calling transcribe(): pip install openai-whisper  import whisper model = whisper.load_model("large") transcript = model.transcribe( word_timestamps=True, audio="toto.mp3" ) for segment in transcript['segments']: print(''.join(f"{word['word']}[{word['start']}/{word['end']}]" for word in segment['words']))  prints:  Toto,[2.98/3.4] I[3.4/3.82] have[3.82/3.96] a[3.96/4.02] feeling[4.02/4.22] we're[4.22/4.44] not[4.44/4.56] in[4.56/4.72] Kansas[4.72/5.14] anymore.[5.14/5.48]
Question: Am calling an endpoint from flask using fetch api from react. I keep getting psycopg2.ProgrammingError: the connection cannot be re-entered recursively I the endpoint call is inside a loop. @app.get("/api/plot-project/<int:plot_id>/<int:project_id>") def check_and_deactivate(plot_id, project_id): with connection: with connection.cursor() as cursor: cursor.execute(PLOT_PROJECT_CHECK, (plot_id, project_id)) data = cursor.fetchall() if len(data) == 0: return "No data found", 404 removed = data.pop(0) if len(data) > 1: for row in data: print(row[0]) cursor.execute(PLOT_PROJECT_DEACTIVATE, ('deleted', row[0], plot_id, project_id)) return { "Remain": removed }, 200  The react fuctions  const handleGet = () => { data.forEach (async (item) => { await getData(item.plotID, item.projectID); }) }  the fetch handle  const getData = async (plotID, projectID) => { fetch(`http://127.0.0.1:5000/api/plot-project/${plotID}/${projectID}`, { method : 'GET', mode: 'no-cors', headers : { 'Content-Type': 'application/json', 'Authorization': `Bearer ${token}` }}) .then(data => data.json()) .then((response) => { console.log('mapping', plotID, "to", projectID) console.log('request succeeded with JSON response', response) }).catch(function (error) { console.log('mapping', plotID, "to", projectID) console.log('no mapping yet') }); } 

Answer: This error happens when you are trying to call the context manager of the same connection, which is already called in the context manager. To fix that issue you have few options:  To use the same connection without context manager and to commit or rollback changes. The code will look like:   try: result = None with connection.cursor() as cursor: cursor.execute(PLOT_PROJECT_CHECK, (plot_id, project_id)) data = cursor.fetchall() if len(data) == 0: result = "No data found", 404 removed = data.pop(0) if len(data) > 1: for row in data: print(row[0]) cursor.execute(PLOT_PROJECT_DEACTIVATE, ('deleted', row[0], plot_id, project_id)) result = { "Remain": removed }, 200 conn.commit() return result except Exception as e: logging.error(f"An error occurred: {str(e)}") conn.rollback() return "Database error", 500   To use the same connection without context manager and to use autocommit. The code will look like:   connection.autocommit = True (you can place that code where you are making the connection) with connection.cursor() as cursor: cursor.execute(PLOT_PROJECT_CHECK, (plot_id, project_id)) data = cursor.fetchall() if len(data) == 0: return "No data found", 404 removed = data.pop(0) if len(data) > 1: for row in data: print(row[0]) cursor.execute(PLOT_PROJECT_DEACTIVATE, ('deleted', row[0], plot_id, project_id)) return { "Remain": removed }, 200   To use connection pool and to enter the context manager of separate connection for every request.
Question: I am using image compression to reduce the image size. When submitting the post request, I am not getting any error, but can't figure out why the images do not get saved. Here is my code: @app.post("/post_ads") async def create_upload_files(title: str = Form(),body: str = Form(), db: Session = Depends(get_db), files: list[UploadFile] = File(description="Multiple files as UploadFile")): for file in files: im = Image.open(file.file) im = im.convert("RGB") im_io = BytesIO() im = im.save(im_io, 'JPEG', quality=50) 

Answer: PIL.Image.open() takes as fp argumnet the following:  fp – A filename (string), pathlib.Path object or a file object. The file object must implement file.read(), file.seek(), and file.tell() methods, and be opened in binary mode.  Using a BytesIO stream, you would need to have something like the below (as shown in client side of this answer): Image.open(io.BytesIO(file.file.read()))  However, you don't really have to use an in-memory bytes buffer, as you can get the actual file object using the .file attribute of UploadFile. As per the documentation:  file: A SpooledTemporaryFile (a file-like object). This is the actual Python file that you can pass directly to other functions or libraries that expect a "file-like" object.  Example - Saving image to disk: # ... from fastapi import HTTPException from PIL import Image @app.post("/upload") def upload(file: UploadFile = File()): try: im = Image.open(file.file) if im.mode in ("RGBA", "P"): im = im.convert("RGB") im.save('out.jpg', 'JPEG', quality=50) except Exception: raise HTTPException(status_code=500, detail='Something went wrong') finally: file.file.close() im.close()  Example - Saving image to an in-memory bytes buffer (see this answer): # ... from fastapi import HTTPException from PIL import Image @app.post("/upload") def upload(file: UploadFile = File()): try: im = Image.open(file.file) if im.mode in ("RGBA", "P"): im = im.convert("RGB") buf = io.BytesIO() im.save(buf, 'JPEG', quality=50) # to get the entire bytes of the buffer use: contents = buf.getvalue() # or, to read from `buf` (which is a file-like object), call this first: buf.seek(0) # to rewind the cursor to the start of the buffer except Exception: raise HTTPException(status_code=500, detail='Something went wrong') finally: file.file.close() buf.close() im.close()  For more details and code examples on how to upload files/images using FastAPI, please have a look at this answer and this answer. Also, please have a look at this answer for more information on defining your endpoint with def or async def.
Question: I create a Plotly Figure instance this way: fig = go.Figure() fig = make_subplots(rows=3, cols=1, shared_xaxes=True, row_width=[0.3, 0.3, 0.4])  Lets assume that now I do not know how many rows and cols the Figure instance has. How can I obtain these values? For example, I expect something like this: rows = fig.get_rows_num() cols = fig.get_cols_num()  I appreciate any help.

Answer: I had the same use case come up! I was happy to find you can do this: rows, cols = fig._get_subplot_rows_columns()
Question: I wrote a Python code using a virtual evn with pip, and I built it with pyinstaller to use it as executable, and it works. Now I'm moving to conda environment to use also geopandas, fiona and gdal. I can run it without any errors, but if I build the code into the .exe, this error raised: Traceback (most recent call last): File "main.py", line 5, in <module> File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module File "openpyxl\__init__.py", line 6, in <module> File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module File "openpyxl\workbook\__init__.py", line 4, in <module> File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module File "openpyxl\workbook\workbook.py", line 9, in <module> File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module File "openpyxl\worksheet\_write_only.py", line 13, in <module> File "openpyxl\worksheet\_writer.py", line 23, in init openpyxl.worksheet._writer ModuleNotFoundError: No module named 'openpyxl.cell._writer' [12248] Failed to execute script 'main' due to unhandled exception!  I tried also to reinstall openpyxl through conda, but nothing changed. The command line to build is: pyinstaller --onefile main_new.spec main.py  and the spec file is: # -*- mode: python ; coding: utf-8 -*- block_cipher = None a = Analysis(['main.py'], pathex=[], binaries=[], datas=[('./inputs/*.csv', 'inputs')], hiddenimports=[ 'openpyxl', 'xlrd', 'xlswriter' ], hookspath=[], hooksconfig={}, runtime_hooks=[], excludes=[], win_no_prefer_redirects=False, win_private_assemblies=False, cipher=block_cipher, noarchive=False) pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher) exe = EXE(pyz, a.scripts, a.binaries, a.zipfiles, a.datas, [], name='DESAT', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, upx_exclude=[], runtime_tmpdir=None, console=True, disable_windowed_traceback=False, target_arch=None, codesign_identity=None, entitlements_file=None )  How can I solve this issue? Thank you!

Answer: The error is referring to 'openpyxl.cell._writer' that is inside openpyxl. in fact, pyinstaller was actually able to find openpyxl. I checked inside, and I found that in the pip environment i was using the 3.0.9 version, while in the conda one I was using the 3.0.10. Downgrading to 3.0.9, no --hidden-import or other needed, it is just working.
Question: I keep getting this error when importing top2vec. TypeError Traceback (most recent call last) Cell In [1], line 1 ----> 1 from top2vec import Top2Vec File ~\AppData\Roaming\Python\Python39\site-packages\top2vec\__init__.py:1 ----> 1 from top2vec.Top2Vec import Top2Vec 3 __version__ = '1.0.27' File ~\AppData\Roaming\Python\Python39\site-packages\top2vec\Top2Vec.py:12 10 from gensim.models.phrases import Phrases 11 import umap ---> 12 import hdbscan 13 from wordcloud import WordCloud 14 import matplotlib.pyplot as plt File ~\AppData\Roaming\Python\Python39\site-packages\hdbscan\__init__.py:1 ----> 1 from .hdbscan_ import HDBSCAN, hdbscan 2 from .robust_single_linkage_ import RobustSingleLinkage, robust_single_linkage 3 from .validity import validity_index File ~\AppData\Roaming\Python\Python39\site-packages\hdbscan\hdbscan_.py:509 494 row_indices = np.where(np.isfinite(matrix).sum(axis=1) == matrix.shape[1])[0] 495 return row_indices 498 def hdbscan( 499 X, 500 min_cluster_size=5, 501 min_samples=None, 502 alpha=1.0, 503 cluster_selection_epsilon=0.0, 504 max_cluster_size=0, 505 metric="minkowski", 506 p=2, 507 leaf_size=40, 508 algorithm="best", --> 509 memory=Memory(cachedir=None, verbose=0), 510 approx_min_span_tree=True, 511 gen_min_span_tree=False, 512 core_dist_n_jobs=4, 513 cluster_selection_method="eom", 514 allow_single_cluster=False, 515 match_reference_implementation=False, 516 **kwargs 517 ): 518 """Perform HDBSCAN clustering from a vector array or distance matrix. 519 520 Parameters (...) 672 Density-based Cluster Selection. arxiv preprint 1911.02282. 673 """ 674 if min_samples is None: TypeError: __init__() got an unexpected keyword argument 'cachedir'  Python version: 3.9.7 (64-bit) Have installed MSBuild No errors when pip installing this package Does anyone know a solution to this problem or experienced a similar problem?

Answer:  UPDATE 12 November 2022:  There is new release (ver. 0.8.29) of hdbscan from 31 Oct. 2022 that fix the issue. See my original answer for more details. Original Answer: It looks like you are using latest (as of 23 Sept 2022) versions of hdbscan and joblib packages available on PyPI. cachedir was removed from joblib.Memory in commit on 2 Feb 2022 as depreciated. The latest version on PyPi is ver. 1.2.0 released on Sep 16, 2022, i.e. it incorporate this change The relevant part of hdbscan source code on GitHub was updated on 16 Sept 2022. Unfortunately the latest (as of 23 Sept 2022) hdbscan release on PyPi is ver. 0.8.28 released on Feb 8, 2022 and still not updated. It still use memory=Memory(cachedir=None, verbose=0) One possible solution is to force using joblib version before cachedir was removed - ver. 1.1.0 as of Oct 7, 2021. However note my edits below.  UPDATE 29 Sept 2022:  There are open issues on hdbscan repo (#563) and (#565). Note there is vulnerability CVE-2022-21797 when using joblib < 1.2.0  UPDATE 12 November 2022:  There is new release (ver. 0.8.29) of hdbscan from 31 Oct. 2022.
Question: After upgrading djangorestframework from djangorestframework==3.13.1 to djangorestframework==3.14.0 the code from rest_framework.serializers import NullBooleanField  Throws  AttributeError: module 'rest_framework.serializers' has no attribute 'NullBooleanField'  Reading the release notes I don't see a deprecation. Where did it go?

Answer: For what it's worth, there's a deprecation warning in the previous version, which also suggests a fix:  The NullBooleanField is deprecated and will be removed starting with 3.14. Instead use the BooleanField field and set allow_null=True which does the same thing.
Question: I'm trying to build and tag artifacts, the environment name gets appended at the end of the release, e.g.: 1.0.0-stg or 1.0.0-sndbx, none of them are PEP-440 compliance, raising the following error message: raise InvalidVersion(f"Invalid version: '{version}'") packaging.version.InvalidVersion: Invalid version: '1.0.0-stg'  Using the packaging library I know I can access the regex by doing: from packaging import version version.VERSION_PATTERN  However, my question is how can I customize the regex rule also to support other environments?

Answer: The original value of the version.VERSION_PATTERN has used regex groups to separate the types of the version. And if you see the pattern, it defines three types of releases: pre-release, post-release, and dev release. v? (?: (?:(?P<epoch>[0-9]+)!)? # epoch (?P<release>[0-9]+(?:\.[0-9]+)*) # release segment (?P<pre> # pre-release [-_\.]? (?P<pre_l>(a|b|c|rc|alpha|beta|pre|preview)) [-_\.]? (?P<pre_n>[0-9]+)? )? (?P<post> # post release (?:-(?P<post_n1>[0-9]+)) | (?: [-_\.]? (?P<post_l>post|rev|r) [-_\.]? (?P<post_n2>[0-9]+)? ) )? (?P<dev> # dev release [-_\.]? (?P<dev_l>dev) [-_\.]? (?P<dev_n>[0-9]+)? )? ) (?:\+(?P<local>[a-z0-9]+(?:[-_\.][a-z0-9]+)*))? # local version  So you should add the stg and sndbx environment names next to the particular group name. In case those values do not present in the list, the version will always be counted as a local version and not a release version. If those are pre-releases, you should add them to the pre_l group. (?P<pre_l>(a|b|c|rc|alpha|beta|pre|preview|stg|sndbx))  In case those are post-release environments, the post_l should be changed, becoming: (?P<post_l>post|rev|r|stg|sndbx)  And if those are dev release environments, the dev_l group should be updated in the same way. (?P<dev_l>dev|stg|sndbx)  Once you have decided on the type of release and set values to the particular groups, you can override the version.VERSION_PATTERN and see the result. from packaging import version version.VERSION_PATTERN = r""" v? (?: (?:(?P<epoch>[0-9]+)!)? # epoch (?P<release>[0-9]+(?:\.[0-9]+)*) # release segment (?P<pre> # pre-release [-_\.]? (?P<pre_l>(a|b|c|rc|alpha|beta|pre|preview)) [-_\.]? (?P<pre_n>[0-9]+)? )? (?P<post> # post release (?:-(?P<post_n1>[0-9]+)) | (?: [-_\.]? (?P<post_l>post|rev|r) [-_\.]? (?P<post_n2>[0-9]+)? ) )? (?P<dev> # dev release [-_\.]? (?P<dev_l>dev|stg|sndbx) [-_\.]? (?P<dev_n>[0-9]+)? )? ) (?:\+(?P<local>[a-z0-9]+(?:[-_\.][a-z0-9]+)*))? # local version """  NOTE: It is not required to add both identifiers of environments into one group. So one of them can be added to another group.
Question: I have the following dataframe: df1 = pd.DataFrame({'ID' : ['T1002.', 'T5006.', 'T5007.'], 'Parent': ['Stay home.', "Stay home.","Stay home."], 'Child' : ['2Severe weather.', "5847.", "Severe weather."]}) ID Parent Child 0 T1002. Stay home. 2Severe weather. 1 T5006. Stay home. 5847. 2 T5007. Stay home. Severe weather.  I want to add the two columns into one and also add the columns' names into the rows. I want also the columns' names to be in bold. Expected outcome: (I cannot make bold the columns names ID, etc)  Joined_columns() 0 ID: T1002. Parent: Stay home. Child: 2Severe weather. 1 ID: T5006. Parent: Stay home. Child: 5847. 2 ID: T5007. Parent: Stay home. Child: Severe weather.  The join is accomplished with the following code: df1_final=df1.stack().groupby(level=0).apply(' '.join).to_frame(0)  But I am not sure how to go to the end. Any ideas?

Answer:  I want to add the two columns into one and also add the columns' name into the rows. I want also the columns names to be in bold.   i would like to have the bold text on Excel.  applying formats to substrings within cells in an excel worksheet requires writing a rich string It is necessary to construct a list of rich strings and then write them out to the worksheet iteratively. The difficulty with this approach is that you have to keep track of the index offset with respect to the rest of the data frame. import pandas as pd import xlsxwriter # create a workbook / worksheet and format book = xlsxwriter.Workbook('text.xlsx') sheet = book.add_worksheet('joined-columns') bold = book.add_format({'bold': True}) # generate the formatted rich string df1 = pd.DataFrame({'ID': ['T1002.', 'T5006.', 'T5007.'], 'Parent': ['Stay home.', "Stay home.","Stay home."], 'Child': ['2Severe weather.', "5847.", "Severe weather."]}) # the list of string fragments is generated using # a double list comprehension. # this flattens the list of lists, where # each inner list is [bold, 'key: ', 'value', ' ']. # the trailing space is removed through `[:-1]` data = df1.apply( lambda row: [a for k, v in row.to_dict().items() for a in [bold, f'{k}: ', v, ' '] ][:-1], axis=1 ).to_list() # write the title & rich strings # the list of string fragments has to be unpacked with `*` # because `write_rich_string` accepts variable # number or arguments after the cell is specified. sheet.write(0, 0, 'Joined_columns()') for idx, cell in enumerate(data): sheet.write_rich_string(idx + 1, 0, *cell) book.close()
Question: I have a toy reinforcement learning project based on the REINFORCE algorithm (here's PyTorch's implementation) that I would like to add batch updates to. In RL, the "target" can only be created after a "prediction" has been made, so standard batching techniques do not apply. As such, I accrue losses for each episode and append them to a list l_losses where each item is a zero-dimensional tensor. I hold off on calling .backward() or optimizer.step() until a certain number of episodes have passed in order to create a sort of pseudo batch. Given this list of losses, how do I have PyTorch update the network based on their average gradient? Or would updating based on the average gradient be the same as updating on the average loss (I seem to have read otherwise elsewhere)? My current method is to create a new tensor t_loss from torch.stack(l_losses), and then run t_loss = t_loss.mean(), t_loss.backward(), optimizer.step(), and zero the gradient, but I'm unsure if this is equivalent to my intents? It's also unclear to me if I should have been running .backward() on each individual loss instead of concatenating them in a list (but holding on the .step() part until the end?

Answer: Gradient is a linear operation so gradient of the average is the same as the average of the gradient. Take some example data import torch a = torch.randn(1, 4, requires_grad=True); b = torch.randn(5, 4);  You could store all the losses and compute the mean as you are doing, a.grad = None x = (a * b).mean(axis=1) x.mean().backward() # gradient of the mean print(a.grad)  Or every iteration to compute the back propagation to get the contribution of that loss to the gradient. a.grad = None for bi in b: (a * bi / len(b)).mean().backward() print(a.grad)  Performance I don't know the internal details of the pytorch backward implementation, but I can tell that (1) the graph is destroyed by default after the backward pass ratain_graph=True or create_graph=True to backward(). (2) The gradient is not kept except for leaf tensors, unless you specify retain_grad; (3) if you evaluate a model twice using different inputs, you can perform the backward pass to individual variables, this means that they have separate graphs. This can be verified with the following code. a.grad = None # compute all the variables in advance r = [ (a * b / len(b)).mean() for bi in b ] for ri in r: # This depends on the graph of r[i] but the graph or r[i-1] # was already destroyed, it means that r[i] graph is independent # of r[i-1] graph, hence they require separate memory. ri.backward() # this will remove the graph of ri print(a.grad)  So if you update the gradient after each episode it will accumulate the gradient of the leaf nodes, that's all the information you need for the next optimization step, so you can discard that loss freeing up resources for further computations. I would expect a memory usage reduction, potentially even a faster execution if the memory allocation can efficiently use the just deallocated pages for the next allocation.
Question: I've got a class that extends ElementTree.Element: import xml.etree.ElementTree as ET from typing import cast class MyElement(ET.Element): def my_method(self): print('OK') xml = '''<test> <sub/> <sub/> </test>''' root: MyElement = cast( MyElement, ET.fromstring(xml, parser=ET.XMLParser(target=ET.TreeBuilder(element_factory=MyElement)))) root.my_method() # this is fine for ch in root: ch.my_method() # PyCharm error message ???  This does work, however the last line is highlighted by PyCharm because it considers ch to be Element, not MyElement. How should I annotate MyElement to make it clear that when I iterate it, I get MyElement instances and not ET.Elements?

Answer: Annotate the __iter__ of MyElement as a method to return Iterator[MyElement]. There is almost no additional runtime overhead. Pycharm and mypy will both pass: from collections.abc import Callable, Iterator class MyElement(ET.Element): def my_method(self): print('OK') __iter__: Callable[..., Iterator['MyElement']]