eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: def video_downloader(video_url_list: List[str], download_folder: str) -> None: """ Download videos from a list of YouTube video URLs. Args: video_urls (List[str]): The list of YouTube video URLs to download. download_folder (str): The folder to save the downloaded videos. """ successful_downloads = 0 valid_urls: List[str] = [] for url in video_url_list: if not url: continue try: with tempfile.TemporaryDirectory() as temp_dir: cleaned_url = clean_youtube_url(url=url) if _get_streams(url=cleaned_url, temp_dir=temp_dir): create_folder(download_folder) _merge_streams( temp_dir=temp_dir, url=url, download_folder=download_folder ) logger.info(f"The video from {url} was downloaded successfully") successful_downloads += 1 valid_urls.append(url) else: logger.warning(f"No valid video found at the URL: {url}") except Exception as e: logger.error( f"Error has occurred while downloading the video from {url}: {e}" ) print(e) if successful_downloads != 0 and (successful_downloads == len(valid_urls)): messagebox.showinfo( title="Finished download", message=f"Your download is complete!.\n\n\t{successful_downloads}/{len(video_url_list[1:])}", )  This is the main function I was using to download YouTube videos using Pytube. It was working fine last night, but this morning it's not. I'm getting the error "pytube.exceptions.RegexMatchError: get_throttling_function_name: could not find match for multiple", which prevents me from downloading any video or audio. Any possible solutions?

Answer: The work around in the official github page issue: https://github.com/pytube/pytube/issues/1684 function_patterns = [ # https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377 # https://github.com/yt-dlp/yt-dlp/commit/48416bc4a8f1d5ff07d5977659cb8ece7640dcd8 # var Bpa = [iha]; # ... # a.C && (b = a.get("n")) && (b = Bpa[0](b), a.set("n", b), # Bpa.length || iha("")) }}; # In the above case, `iha` is the relevant function name r'a\.[a-zA-Z]\s*&&\s*\([a-z]\s*=\s*a\.get\("n"\)\)\s*&&.*?\|\|\s*([a-z]+)', r'\([a-z]\s*=\s*([a-zA-Z0-9$]+)(\[\d+\])?\([a-z]\)',]
Question: code doesn't run correctly ctrl + B and search on google, and also I searched on Youtube, Google and I tried several ways. there was some solutions but it didn't work. there is no error but unfortunately, my code doesn't run

Answer: Download the portable sublime3, go to: Tools | Build Systems | Python. With "Build" you can run your code.  Crtl + B shows your python version, what is in your system environment variable defined:  I would anyway suggest Thonny
Question: I am trying to write code to get the direct urls of images in a public Google Drive folder and embed them. Until a few minutes ago everything has been working and using the uc? trick gives the raw image so I can embed it. However now it is sending a download, even when I use &export=view. Here is my code: key = "supersecretgooglecloudconsolekey" import re import requests def get_raw_image_links(folder_link): # Extract folder id folder_id = re.findall(r"/folders/([^\s/]+)", folder_link)[0] # Google cloud console request to get links api_url = f"https://www.googleapis.com/drive/v3/files?q='{folder_id}'+in+parents+and+mimeType+contains+'image/'&key={key}" response = requests.get(api_url).json() raw_image_links = [] # Iterate over the files and retrieve raw image links for file in response['files']: raw_link = f"https://drive.google.com/uc?id={file['id']}" print(raw_link) raw_image_links.append(f'<a href="{raw_link}"> <img src="{raw_link}" /> </a>') return ''.join(raw_image_links) folder_link = "https://drive.google.com/drive/u/0/folders/**********" image_links = get_raw_image_links(folder_link)  For example, this works: https://drive.google.com/uc?id=15bwLlBuPc4O8LXwxjsBJe_ctyPl8Tll2 But this doesn't for some reason and instead sends a download: https://drive.google.com/uc?id=1Gz4YrH2tefSVxKRvv6gnqnY3tk0Obm0T How do I fix this? Also, it would be helpful if anyone knows how to scrape the raw image link from the viewing page of the image itself https://drive.google.com/file/d/15bwLlBuPc4O8LXwxjsBJe_ctyPl8Tll2/view Thank you!

Answer: In your situation, how about changing the endpoint? When your showing script is modified, please modify raw_link as follows. From: raw_link = f"https://drive.google.com/uc?id={file['id']}"  To: raw_link = f"https://drive.google.com/thumbnail?id={file['id']}&sz=w1000" # Please modify w1000 to your actual situation.   By this change, the image files are converted to PNG format. I thought that this might be one of several solutions. From your question and showing script, I guessed that your folder of folder_link = "https://drive.google.com/drive/u/0/folders/**********" might be publicly shared. In that case, I thought that the above modification might be able to be used.  Reference:  Permanent links to thumbnails in Google Drive API
Question: I currently have a fastapi deployed with uvicorn that starts a thread on initialisation (among other things) using threading. This thread is infinite (it's a routine that updates every x seconds). Before I updated to python 3.10, everything was working fine, everytime I changed the code, the server would detect change and reload, killing and creating a new thread at init. But now, when I modify my code, the server detects change and try to reload but the created thread isn't killed (print still continue to flow in the console) refraining the server to fully reload. my print from my thread WARNING: StatReload detected changes in 'app\api.py'. Reloading... INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [3736] my print from my thread my print from my thread ...  This works the same way if I ctrl+C in the console. The thread stays alive My solution for the moment is to kill PIDeverytime I want to refresh but that's a bit annoying. I tried to get back to python 3.7.9 but the problem remains. I also tried to implement atexit and manually kill the process but it didn't work. Any lead on how to properly handle this ?

Answer: Fixed by downgrading uvicorn to an older version (0.19.0 works).
Question: I am trying to connect to my redshift cluster using Sqlalchemy in a linux environment, but i am facing the following issue.  from sqlalchemy import create_engine import pandas as pd conn = create_engine('postgresql://connection string') data_frame = pd.read_sql_query("SELECT * FROM schema.table", conn)    sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject) unrecognized configuration parameter "standard_conforming_strings"  I really dont understand what the issue is. It works perfectly fine in Windows. PS: Not sure if this makes any difference but i have installed psycopg2-binary on the linux machine in contrast to psycopg2 on windows. EDIT 1 .The version of pyscopg2 is windows is 2.9.3 while the version of pyscopg2-binary in linux is 2.9.6  The version of Sqlalchemy in windows is 1.4.39 while in linux is 2.0.16 

Answer: I have figured out the answer based on the comment by @Adrian . Just changed the version of Sql alchemy on the linux env to what i have on Windows and it works now.
Question: I have a data file in json format downloaded from another system. It can be observed that it contains one row with all the column names as shown as keys in the json string with its values. Metadata for this one row contains another json string. { "createdTime": 1625433352096, "lastUpdatedTime": 1664815053126, "rootId": 5710030000009051, "parentId": 0000010904715472, "parentExtId": "J802-RFPS-2F-1004", "externalId": "J802-RFPS-003FT1004A", "name": "J802-RFPS-003FT2224A", "description": "113FT1004A, RFP5 CRDE CHG PMP TO TN 1", "dataSetId": 4833410000089553, "metadata": { "Catalog Profile": "16", "Cost Center": "TNK7222414", "Installation of equipment allowed": "Yes", "Location": "112-SCU", "Location and Account Assignment": "11223355", "PP Work Center": "10000420", "Planner Group": "CRD", "Plant Section": "CE", "SAP Object No": "01649", "Single Equipment Installation": "Yes", "Type": "I_CVL", "Work Center": "9876542" }, "source": "SAP PM", "id": 8972234567977 }  I am using the following code in python to read it as dataframe. import json from pathlib import Path import pandas as pd json_file_path = r'D:\asset-897428209692977.json' # path to file p = Path(json_file_path) # read json with p.open('r', encoding='utf-8') as f: data = json.loads(f.read()) # create dataframe df2 = pd.json_normalize(data)  The output has one row but the dataframe splits the metadata into multiple columns such as metadata.planner, metadata.PM, metadata.ID...... Is there another way to keep it as one row so that when some keys are missing in certain records, it doesn't find itself missing or removed in this json parsing. Please advise the best approach

Answer: I don't think there is any direct way to do this in a direct manner. However, you can still include metadata into the database manually, and then add metadata manually as a single column. metadata = data.pop('metadata') df2 = pd.json_normalize(data) df2['metadata'] = [metadata]  However I'm not able to test the code on the data you've provided, so I am not sure it'll work without without any issues. Both original and modified versions do return the same error - "json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 108 (char 107)" while I'm trying to run them, and I'm not sure why does the error happen.
Question: I apologise in advance, I may just not understand convolution. I'm struggling to reconcile the results using scipy.ndimage.convolve with what I get attempting to do it by hand. For the example in the documentation: import numpy as np a = np.array([[1, 2, 0, 0], [5, 3, 0, 4], [0, 0, 0, 7], [9, 3, 0, 0]]) k = np.array([[1,1,1],[1,1,0],[1,0,0]]) from scipy import ndimage ndimage.convolve(a, k, mode='constant', cval=0.0) array([[11, 10, 7, 4], [10, 3, 11, 11], [15, 12, 14, 7], [12, 3, 7, 0]])  However I would expect the result to be:  ([[1, 8, 5, 0], [8, 11, 5, 4], [8, 17, 10, 11], [9, 12, 10, 7]])  For example for the top left value: 1×0 (extended beyond the input) 1×0 (extended beyond the input) 1×0 (extended beyond the input) 1×0 (extended beyond the input) 1×1 0×2 1×0 (extended beyond the input) 0×5 0×3 ___ =1  I don't see how it can be 11 What am I misunderstanding about convolution, arrays, or what scipy is doing here?

Answer: The operation you're performing by hand is cross correlation, not convolution. The process of convolution is similar but "flips" the kernel. You can show that your expected result can be obtained by first flipping your kernel which then gets unflipped during covolution: > ndimage.convolve(a, np.flip(k), mode='constant', cval=0.0) array([[ 1, 8, 5, 0], [ 8, 11, 5, 4], [ 8, 17, 10, 11], [ 9, 12, 10, 7]])  A bit more information here: https://cs.stackexchange.com/questions/11591/2d-convolution-flipping-the-kernel
Question: I create the Google API Resource class for a specific type (in this case blogger). from googleapiclient.discovery import Resource, build def get_google_service(api_type) -> Resource: credentials = ... return build(api_type, 'v3', credentials=credentials) def blog_service(): return get_google_service('blogger') def list_blogs(): return blog_service().blogs()  The problem arises when using the list_blogs function. Since I am providing a specific service name, I know that the return value of blog_service has a blogs method, but my IDE doesn't recognize it. Is there a way to annotate the blog_service function (or any other part of the code) to help my IDE recognize the available methods like blogs?

Answer: The problem here is that the Resource class dynamically sets arbitrary attributes/methods based on what collections the underlying API provides. In your case apparently there is a blogs collection, which means the blogs method is dynamically constructed on that Resource object. Expecting static type annotations, when your types are created dynamically is a tall order. (I assume this is one of the reasons the maintainers of google-api-python-client do not even bother with type-hinting in that package.) But depending on your goals with those functions, you might improve your IDE experience by using protocols. If you know that your blog_service function returns an object that has a blogs method, you can define a corresponding protocol. The problem is of course kicked down the road because whatever that blogs method returns may also have arbitrary methods. But depending on if this is important to you, you can apply the same principle for that again. The upside is that Resource itself actually seems to have very few public methods, essentially just close and the context manager protocol. So you could emulate that in some sort of base protocol and use inheritance to construct different resource-protocols. Here is an example to illustrate: from typing import Any, Protocol, Self from googleapiclient.discovery import build # type: ignore[import] class ResourceProtocol(Protocol): def close(self) -> None: ... def __enter__(self) -> Self: ... def __exit__(self, *args: Any) -> None: ... class BloggerProtocol(ResourceProtocol): def blogs(self) -> Any: ... def get_google_service(api_type: str) -> Any: credentials = ... return build(api_type, 'v3', credentials=credentials) def blog_service() -> BloggerProtocol: return get_google_service('blogger') # type: ignore[no-any-return] def list_blogs() -> Any: return blog_service().blogs()  (Note that those are all literal ellipses .... Also, if you are on Python <3.11, you should be able to import Self from typing_extensions instead.) Now your IDE should be able to detect that the object returned by blog_service has a blogs method and can be used as a context manager (using with). Notice that I annotated the return type of blogs as Any because I don't know what type it is supposed to return.
Question: I'm getting this very strange error when trying to follow the following exercise on using the corr() method in Python https://www.geeksforgeeks.org/python-pandas-dataframe-corr/ Specifically, when I try to run the following code: df.corr(method ='pearson') The error message offers no clue. I thought the corr() method was supposed to automatically ignore strings and empty values etc. Traceback (most recent call last): File "<pyshell#6>", line 1, in <module> df.corr(method='pearson') File "C:\Users\d.o\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py", line 10059, in corr mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False) File "C:\Users\d.o\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py", line 1838, in to_numpy result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value) File "C:\Users\d.o\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\internals\managers.py", line 1732, in as_array arr = self._interleave(dtype=dtype, na_value=na_value) File "C:\Users\d.o\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\internals\managers.py", line 1794, in _interleave result[rl.indexer] = arr ValueError: could not convert string to float: 'Avery Bradley' 

Answer: When I try to replicate this behavior, the corr() method works OK but spits out a warning (shown below) that warns that the ignoring of non-numeric columns will be removed in the future. Perhaps the future has arrived? I've got pandas version 1.5.3. You may need to just specify which columns to use--which is actually a better way to do it rather than rely on pd to do this for you. You can do that by just supplying a list of the columns of interest as an index (shown below.) In [1]: import pandas as pd In [2]: data = {'name': ['bob', 'cindy', 'tom'], ...: 'x' : [ 1, 2, 3 ], ...: 'y' : [ 6.5, 8.9, 12.0]} In [3]: df = pd.DataFrame(data) In [4]: df Out[4]: name x y 0 bob 1 6.5 1 cindy 2 8.9 2 tom 3 12.0 In [5]: df.describe() Out[5]: x y count 3.0 3.000000 mean 2.0 9.133333 std 1.0 2.757414 min 1.0 6.500000 25% 1.5 7.700000 50% 2.0 8.900000 75% 2.5 10.450000 max 3.0 12.000000 In [6]: df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 3 entries, 0 to 2 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 name 3 non-null object 1 x 3 non-null int64 2 y 3 non-null float64 dtypes: float64(1), int64(1), object(1) memory usage: 200.0+ bytes In [7]: df.corr(method='pearson') <ipython-input-7-432dd9d4238b>:1: FutureWarning: The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning. df.corr(method='pearson') Out[7]: x y x 1.000000 0.997311 y 0.997311 1.000000 In [8]: df[['x', 'y']].corr(method='pearson') Out[8]: x y x 1.000000 0.997311 y 0.997311 1.000000
Question: I have this image    I need to find an empty area that can fit this shape    so that the end result is something like this  

Answer: Here is a simple yet naive solution to this problem. It uses 2D convolution as mentioned by UnquoteQuote. import random import numpy as np import scipy.signal as sig import matplotlib.pyplot as plt import cv2 # load images image = (cv2.imread('image.jpg').mean(axis=2) > 127).astype(np.float32) shape = (cv2.imread('shape.jpg').mean(axis=2) > 127).astype(np.float32) # perform 2D convolution conv = sig.convolve2d(image, shape, mode='valid') solutions = np.where(conv == 0) # draw some solutions plt.figure(figsize=(16, 4)) for i in range(4): r = random.randint(0, solutions[0].shape[0] - 1) x, y = solutions[0][r], solutions[1][r] solution_plot = np.zeros((*image.shape, 3)) solution_plot[:, :, 0] = image solution_plot[x:x + shape.shape[0], y:y + shape.shape[1], 1] = shape plt.subplot(1, 4, i + 1) plt.imshow(solution_plot) plt.show()  Example results:  This algorithm finds all possible solutions. If you only need one, you can optimize it so that it gets a random (x, y) point and perform dot product of the shape and a cropped image area [x:x+shape_width, y:y+shape_height] to check if there is a space until you find the right point. This can be done for example like this: while True: x = random.randint(0, image.shape[0] - shape.shape[0]) y = random.randint(0, image.shape[1] - shape.shape[1]) if np.sum(shape*image[x:x + shape.shape[0], y:y + shape.shape[1]]) == 0: break # x, y is the solution  Compared to convolution this one is much faster (but it depends on the number of the solutions):  convolution: 6.65 s ± 21.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) random search: 1.31 ms ± 31.2 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
Question: I read on https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/chatgpt?pivots=programming-language-chat-completions: openai.api_version = "2023-05-15"  and on https://learn.microsoft.com/en-us/answers/questions/1193969/how-to-integrate-tiktoken-library-with-azure-opena: openai.api_version = "2023-03-15-preview"  This makes me wonder: How can I select the proper openai.api_version? Does that depend on my Azure OpenAI instance or deployed models or which features I use in my Python code? Or something else? I couldn't find the info in my deployed models: 

Answer: The API Version property depends on the method you are calling in the API: all methods are not supported in all API versions. Details are listed here: https://learn.microsoft.com/en-US/azure/cognitive-services/openai/reference And preview API lifecycle is described here: https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation => check those pages for up-to-date references As of March 7th, 2024: Example: "completions" endpoint is available in the following versions (ordered by date):  2024-02-15-preview 2023-12-01-preview (retiring April 2, 2024) 2023-09-01-preview (retiring April 2, 2024) 2023-08-01-preview (retiring April 2, 2024) 2023-07-01-preview (retiring April 2, 2024) 2023-06-01-preview (still supported, due to DALL-E 2) 2023-05-15 2023-03-15-preview (retiring April 2, 2024) 2022-12-01  But "chat completions" endpoint is available only in the following versions (ordered by date):  2024-02-15-preview 2023-12-01-preview (retiring April 2, 2024) 2023-09-01-preview (retiring April 2, 2024) 2023-08-01-preview (retiring April 2, 2024) 2023-07-01-preview (retiring April 2, 2024) 2023-06-01-preview (still supported, due to DALL-E 2) 2023-05-15 2023-03-15-preview (retiring April 2, 2024) Because basically it was not offered in the initial API.  Generally, use the latest "not preview" version for production when possible, as the preview versions might be retired on a more frequent basis.
Question: I am new to SQLAlchemy and I see that in the documentation the older version (Column) can be swapped directly with the newer "mapped_column". Is there any advantage to using mapped_column over Column? Could you stick to the older 'Column'?

Answer: I think originally Column was used in the lower "core"/sqlalchemy.sql layer AND the higher ORM layer. This created a conflict of purpose. So mapped_column now supersedes Column when using the ORM layer to add more functionality that can't be used by the core layer. The core layer will keep using Column. So I think it is just meant to help you do more faster or more succinctly with the ORM. There is a blurb about them titled "mapped_column() supersedes the use of Column()" below declarative-table-with-mapped-column. Here are some basic examples, using postgresql. See SQL output at end. class Base(DeclarativeBase): pass class Controller(Base): __tablename__ = "controllers" id: Mapped[int] = mapped_column(primary_key=True) name: Mapped[str] = mapped_column() # Example 1 index: Mapped[int] # Example 2 configured: Mapped[Optional[bool]] # Example 3 setup_mode: Mapped[bool] # Example 4 created_at = Column(DateTime(timezone=True)) # Example 5  Example 1 The column type is derived from the type hint, VARCHAR is derived from str in this case. name: Mapped[str] = mapped_column()  Example 2 When mapped_column would be empty it can be left out entirely and this still works, ie. INTEGER is derived from int. index: Mapped[int]  Example 3 When Optional from typing is used then a column will allow NULL, ie. nullable=True. configured: Mapped[Optional[bool]]  Example 4 When Optional is NOT used then a column will not allow NULL, ie. nullable=False. setup_mode: Mapped[bool]  Example 5 Column can still be used alongside mapped_column without using type hints at all. created_at = Column(DateTime(timezone=True))  Example of type checking Running this code through mypy will produce an error similar to error: "Controller" has no attribute "unknown_attribute" [attr-defined]  with Session(engine) as session: controller = session.scalars(select(Controller).limit(1)).first() if controller is not None: assert controller.created_at assert controller.unknown_attribute  Final CREATE TABLE output CREATE TABLE controllers ( id SERIAL NOT NULL, name VARCHAR NOT NULL, index INTEGER NOT NULL, configured BOOLEAN, setup_mode BOOLEAN NOT NULL, created_at TIMESTAMP WITH TIME ZONE, PRIMARY KEY (id) )  Some "Why"s  allow more orm specific functionality that does not make sense to be in Column() reduce boilerplate code  int, str, datetime, Optional, etc. are available from python without needing sqlalchemy imports In cases where the type can be derived from the typehint and no special configuration is necessary the entire column/mapped_column definition can be left out, ie. index: Mapped[int]   allow type checkers to better check types  the checks can be expanded by using something like the data-class integration but is beyond this question