eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a very powerful bot in discord (discord.py, PYTHON) and it can play music in voice channels. It gets the music from youtube (youtube_dl). It worked perfectly before but now it doesn't want to work with any video. I tried updating youtube_dl but it still doesn't work I searched everywhere but I still can't find a answer that might help me. This is the Error: Error: Unable to extract uploader id After and before the error log there is no more information. Can anyone help? I will leave some of the code that I use for my bot... The youtube setup settings: youtube_dl.utils.bug_reports_message = lambda: '' ytdl_format_options = { 'format': 'bestaudio/best', 'outtmpl': '%(extractor)s-%(id)s-%(title)s.%(ext)s', 'restrictfilenames': True, 'noplaylist': True, 'nocheckcertificate': True, 'ignoreerrors': False, 'logtostderr': False, 'quiet': True, 'no_warnings': True, 'default_search': 'auto', 'source_address': '0.0.0.0', # bind to ipv4 since ipv6 addresses cause issues sometimes } ffmpeg_options = { 'options': '-vn', } ytdl = youtube_dl.YoutubeDL(ytdl_format_options) class YTDLSource(discord.PCMVolumeTransformer): def __init__(self, source, *, data, volume=0.5): super().__init__(source, volume) self.data = data self.title = data.get('title') self.url = data.get('url') self.duration = data.get('duration') self.image = data.get("thumbnails")[0]["url"] @classmethod async def from_url(cls, url, *, loop=None, stream=False): loop = loop or asyncio.get_event_loop() data = await loop.run_in_executor(None, lambda: ytdl.extract_info(url, download=not stream)) #print(data) if 'entries' in data: # take first item from a playlist data = data['entries'][0] #print(data["thumbnails"][0]["url"]) #print(data["duration"]) filename = data['url'] if stream else ytdl.prepare_filename(data) return cls(discord.FFmpegPCMAudio(filename, **ffmpeg_options), data=data)  Approximately the command to run the audio (from my bot): sessionChanel = message.author.voice.channel await sessionChannel.connect() url = matched.group(1) player = await YTDLSource.from_url(url, loop=client.loop, stream=True) sessionChannel.guild.voice_client.play(player, after=lambda e: print( f'Player error: {e}') if e else None) 

Answer: This is a known issue, fixed in Master. For a temporary fix, python3 -m pip install --force-reinstall https://github.com/yt-dlp/yt-dlp/archive/master.tar.gz  This installs tha master version. Run it through the command-line yt-dlp URL  where URL is the URL of the video you want. See yt-dlp --help for all options. It should just work without errors. If you're using it as a module, import yt_dlp as youtube_dl  might fix your problems (though there could be API changes that break your code; I don't know which version of yt_dlp you were using etc).
Question: How to tell whether an argument in click is coming from the user or is the default value? For example: import click @click.command() @click.option('--value', default=1, help='a value.') def hello(value): print(value) if __name__ == "__main__": hello()  Now if I run python script.py --value 1, the value is now coming from the user input as opposed to the default value (which is set to 1). Is there any way to discern where this value is coming from?

Answer: You can use Context.get_parameter_source to get what you want. This returns an enum of 4 possible values (or None if the value does not exist), you can then use them to decide what you want to do. COMMANDLINE - The value was provided by the command line args. ENVIRONMENT - The value was provided with an environment variable. DEFAULT - Used the default specified by the parameter. DEFAULT_MAP - Used a default provided by :attr:`Context.default_map`. PROMPT - Used a prompt to confirm a default or provide a value.  import click from click.core import ParameterSource @click.command() @click.option('--value', default=1, help='a value.') def hello(value): parameter_source = click.get_current_context().get_parameter_source('value') if parameter_source == ParameterSource.DEFAULT: print('default value') elif parameter_source == ParameterSource.COMMANDLINE: print('from command line') # other conditions go here print(value) if __name__ == "__main__": hello()  In this case, the value is taken from default and can be seen in the output below. Output default value 1
Question: I'm upserting data in snowflake table by creating a Temp Table (from my dataframe) and then merging it to my Table. But is there a more efficient way of achieving it ? Like merging directly the dataframe on snowflake table without a temp Table ? Because I will do it on several tables having a few thousant rows. My Code:  import pandas as pd from sqlalchemy import create_engine from snowflake.connector.pandas_tools import pd_writer engine = create_engine('snowflake://{user}:{password}@{account_identifier}/{database_name}/{schema_name}?warehouse={warehouse_name}&role={role_name}'.format( user='user', password=os.environ['SNOWFLAKE_PASSWORD'] , account_identifier='account_identifier', database_name='DB_NAME', schema_name='SHCEMA_NAME', warehouse_name='WH', role_name='ADMIN' ) ) conn=engine.connect() temp_table_name='source_table' df=pd.DataFrame({'id':[1,2,3],'description':['a','b','c']}) #create temp table in snowflake res_sql=df.to_sql(temp_table_name.lower(), engine, if_exists='replace',index=False, method=pd_writer, schema='SCHEMA_NAME') #MERGE TEMP TABLE TO EXISTING TABLE conn.cursor().execute( ''' MERGE INTO target_table USING source_table ON target_table.id = source_table.id WHEN MATCHED THEN UPDATE SET target_table.description = source_table.description WHEN NOT MATCHED THEN INSERT (ID, description) VALUES (source_table.id, source_table.description); ''' ) #Drop temp table conn.cursor().execute("DROP TABLE IF EXISTS DB_NAME.SCHEMA_NAME.source_table") 

Answer: Using snowflake.snowpark.Table.merge:  Merges this Table with DataFrame source on the specified join expression and a list of matched or not-matched clauses, and returns a MergeResult, representing the number of rows inserted, updated and deleted by this merge action.  Standalone sample(table target exists at Snowflake): CREATE OR REPLACE TABLE target(key TEXT, value TEXT); INSERT INTO target VALUES (10, 'old'), (10, 'too_old'), (11, 'old'); SELECT * FROM target;  Python code(here wrapped with WITH PROCEDURE ... CALL for simplicity): WITH proc AS PROCEDURE() RETURNS TEXT LANGUAGE PYTHON RUNTIME_VERSION = '3.8' PACKAGES = ('snowflake-snowpark-python') HANDLER = 'main' AS $$ from snowflake.snowpark.functions import when_matched, when_not_matched def main(session): source = session.create_dataframe([(10, "new"), (12, "new"), (13, "old")], schema=["key", "value"]) target = session.table("target") return target.merge(source, (target["key"] == source["key"]) & (target["value"] == "too_old"), [when_matched().update({"value": source["value"]}), when_not_matched().insert({"key": source["key"]})]) $$ CALL proc();  Output: MergeResult(rows_inserted=2, rows_updated=1, rows_deleted=0)    Related:  Snowpark API Snowpark Snowflake - invoking Python code without creating UDF/Stored Procedure - for this ad-hoc sample, normally called from client tool
Question: A data frame below and I want to calculate the intervals of months under the names. Lines so far: import pandas as pd from io import StringIO import numpy as np csvfile = StringIO( """Name Year - Month Score Mike 2022-11 31 Mike 2022-11 136 Lilly 2022-11 23 Lilly 2022-10 44 Kate 2023-01 1393 Kate 2022-10 2360 Kate 2022-08 1648 Kate 2022-06 543 Kate 2022-04 1935 Peter 2022-04 302 David 2023-01 1808 David 2022-12 194 David 2022-09 4077 David 2022-06 666 David 2022-03 3362""") df = pd.read_csv(csvfile, sep = '\t', engine='python') df['Year - Month'] = pd.to_datetime(df['Year - Month'], format='%Y-%m') df['Interval'] = (df.groupby(['Name'])['Year - Month'].transform(lambda x: x.diff())/ np.timedelta64(1, 'M')) df['Interval'] = df['Interval'].replace(np.nan, 1).astype(int)  But the output seems something wrong (not calculating right). Where has this gone wrong, and how can I correct it?  Name Year - Month Score Interval 0 Mike 2022-11 31 1 <- shall be 0 1 Mike 2022-11 136 0 2 Lilly 2022-11 23 1 3 Lilly 2022-10 44 1 <- shall be 0 4 Kate 2023-01 1393 1 <- shall be 3 5 Kate 2022-10 2360 3 <- shall be 2 6 Kate 2022-08 1648 2 7 Kate 2022-06 543 2 8 Kate 2022-04 1935 2 <- shall be 0 9 Peter 2022-04 302 1 <- shall be 0 10 David 2023-01 1808 1 <- shall be 1 11 David 2022-12 194 1 <- shall be 3 12 David 2022-09 4077 2 <- shall be 3 13 David 2022-06 666 3 14 David 2022-03 3362 3 <- shall be 0 

Answer: You need to difference with next value instead of previous value. You can do so by setting -1 in diff(). ... df['Interval'] = df.groupby(['Name'])['Year - Month'].transform(lambda x: x.diff(-1)) / np.timedelta64(1, 'M') df['Interval'] = df['Interval'].fillna(0).round().astype(int)  Result:  Name Year - Month Score Interval 0 Mike 2022-11-01 31 0 1 Mike 2022-11-01 136 0 2 Lilly 2022-11-01 23 1 3 Lilly 2022-10-01 44 0 4 Kate 2023-01-01 1393 3 5 Kate 2022-10-01 2360 2 6 Kate 2022-08-01 1648 2 7 Kate 2022-06-01 543 2 8 Kate 2022-04-01 1935 0 9 Peter 2022-04-01 302 0 10 David 2023-01-01 1808 1 11 David 2022-12-01 194 3 12 David 2022-09-01 4077 3 13 David 2022-06-01 666 3 14 David 2022-03-01 3362 0
Question: How do I find the msmallest surrounding rectangle (which is possibly rotated) of a set of 2D points in Shapely?

Answer: To create the smallest surrounding rectangle in Shapely, first construct a MultiPoint from a sequence of points then use the minimum_rotated_rectangle property (which is in the BaseGeometry class). from shapely.geometry import MultiPoint, Polygon points = [(0, 0), (2, 2), (10, 4), (5, 5), (8, 8)] # create a minimum rotated rectangle containing all the points polygon = MultiPoint(points).minimum_rotated_rectangle print(polygon)  Output: POLYGON ((2.9999999999999996 -2.9999999999999996, 10.999999999999998 4.999999999999998, ...  The example set of points are displayed below in red and the bounding box is in blue.  If want to create an envelope around the points that is the smallest rectangle (with sides parallel to the coordinate axes) containing all the points then call the envelope property on the object. polygon = MultiPoint(points).envelope
Question: For example, from https://docs.python-requests.org/en/latest/api/#requests.cookies.RequestsCookieJar.set: set(name, value, **kwargs) Dict-like set() that also supports optional domain and path args in order to resolve naming collisions from using one cookie jar over multiple domains.  Where can I find information about what other arguments the function takes as **kwargs? I mean these arguments, domain, path, expires, max_age, secure, httponly. It is not documented there! All other functions are like this, I got confused what to pass as parameters. In php.net they describe all parameters properly. Where can I find all parameters that are hidden behind **kwargs?

Answer: In my experience reading the source code for many open source libraries solves this problem. For the example you posted the source code is the following: def set(self, name, value, **kwargs): """Dict-like set() that also supports optional domain and path args in order to resolve naming collisions from using one cookie jar over multiple domains. """ # support client code that unsets cookies by assignment of a None value: if value is None: remove_cookie_by_name( self, name, domain=kwargs.get("domain"), path=kwargs.get("path") ) return if isinstance(value, Morsel): c = morsel_to_cookie(value) else: c = create_cookie(name, value, **kwargs) self.set_cookie(c) return c  For python kwargs are viewed as a dictionary (that's what ** does). In this case the set function uses the "domain" and "path" directly. However, there is another function that takes **kwargs. This is the main purpose of using kwargs instead of fixing the arguments. If we dive into the source code of create_cookie we can see which keyword arguments are valid. def create_cookie(name, value, **kwargs): """Make a cookie from underspecified parameters. By default, the pair of `name` and `value` will be set for the domain '' and sent on every request (this is sometimes called a "supercookie"). """ result = { "version": 0, "name": name, "value": value, "port": None, "domain": "", "path": "/", "secure": False, "expires": None, "discard": True, "comment": None, "comment_url": None, "rest": {"HttpOnly": None}, "rfc2109": False, } badargs = set(kwargs) - set(result) if badargs: raise TypeError( f"create_cookie() got unexpected keyword arguments: {list(badargs)}" ) result.update(kwargs) result["port_specified"] = bool(result["port"]) result["domain_specified"] = bool(result["domain"]) result["domain_initial_dot"] = result["domain"].startswith(".") result["path_specified"] = bool(result["path"]) return cookielib.Cookie(**result)  In this case the only allowed keywords are the ones described in the results dictionary.
Question: I have two same length timeline-like series and I want to merge the parts that appear first while not overlapping. For Example: long = [0,0,1,1,1,1,0,0,0,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,0] short = [0,0,0,1,1,1,1,0,1,1,1,1,1,1,0,0,0,1,1,1,0,0,1,1,1]  wanted:  result = [0,0,1,1,1,1,0,0,1,1,1,1,1,1,0,0,0,1,1,1,0,0,1,1,1] # ------- ----------- --------------- # from long from short from short  I have a chunked iterative solution: def np_shift(arr, shift=1): arr = arr.astype(float) arr = np.roll(arr, shift) arr[:shift] = np.nan return arr def split_arr_to_block(arr, index): return np.split(arr.transpose(), index) def calc(zero, long, short): p_block = (zero!=(np_shift(zero))).cumsum() split_index = np.unique(p_block, return_index=True)[1][1:] blocks = split_arr_to_block(np.array([long, short]), split_index) def select(arr): dT = arr.transpose() return np.where( (dT[0,:][0]==1), dT[0,:], dT[1,:] ) result = np.array([]) for sec in blocks: result = np.append(result, select(sec)) return result long = np.array([0,0,1,1,1,1,0,0,0,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,0]) short = np.array([0,0,0,1,1,1,1,0,1,1,1,1,1,1,0,0,0,1,1,1,0,0,1,1,1]) zero = np.where( ((long==0) & (short==0)), 1, 0 ) result = calc(zero, long, short)  This solutiontake about 46ms with a 160000+ dataset, is there a faster solution？

Answer: I do not think any pure-Numpy code can efficiently compute this. Thus, this is the perfect use-case for Numba or Cython. You can solve this using a few simple nested loops for loop that will be compiled to a very-fast native code: import numpy as np import numba as nb long = np.array([0,0,1,1,1,1,0,0,0,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,0]) short = np.array([0,0,0,1,1,1,1,0,1,1,1,1,1,1,0,0,0,1,1,1,0,0,1,1,1]) # Assume the input arrays are contiguous and contains 32-bit/64-bit integers @nb.njit(['(int32[::1], int32[::1])', '(int64[::1], int64[::1])']) def compute(a, b): assert a.size == b.size n = a.size r = np.empty_like(a) i = 0 while i < n: if a[i] + b[i] == 0: while i < n and a[i] + b[i] == 0: r[i] = 0 i += 1 elif a[i] != 0: while i < n and a[i] + b[i] != 0: r[i] = a[i] i += 1 else: while i < n and a[i] + b[i] != 0: r[i] = b[i] i += 1 return r result = compute(long, short)  Note a[i] + b[i] == 0 is a fast equivalent to a[i] == 0 and b[i] == 0.  Performance results Here are results with two random arrays containing 160000 integers (32-bit) on my i5-9600KF processor: Initial pure-Numpy code: 4153.57 ms Provided Numba code: 0.51 ms  The provided code is thus 8144 times faster!
Question: I am trying to open a C program in a python script running on a Heroku dyno. The Python script works fine locally, but on the dyno it says that the executable cannot be found. The line to run the program in Python is: proc = subprocess.Popen(["./backend/test-print"], stdout=subprocess.PIPE, stderr=subprocess.PIPE), where backend is the folder that contains the program test-print and the Python script. I run the Python scripy from the folder root so it finds the script just fine. The Heroku logs say: FileNotFoundError: [Errno 2] No such file or directory: './backend/test-print'. If I run bash on the dyno and try to run the program manually, it gives the same error: Running bash on ⬢ ******** ... up, run.8123 (Eco) ~ $ cd backend ~/backend $ ls server.py test-print ~/backend $ ./test-print bash: ./test-print: No such file or directory  Any ideas? Thanks in advance. I built the program test-print on my local machine (not the one I am running the program on). I tried putting the program in the root /app folder to see if it would be found then, but that did not work. EDIT: I should also add that when I cat test-print, it finds the file fine and prints its contents. EDIT: type test-print outputs ~/backend $ type test-print bash: type: test-print: not found  EDIT: ~/backend $ ls -laQ total 28 drwx------ 2 u5587 dyno 4096 Feb 20 16:16 "." drwx------ 5 u5587 dyno 4096 Feb 20 17:08 ".." -rw------- 1 u5587 dyno 520 Feb 20 16:16 "server.py" -rwx------ 1 u5587 dyno 16176 Feb 20 16:16 "test-print" ~/backend $ ls -laq total 28 drwx------ 2 u5587 dyno 4096 Feb 20 16:16 . drwx------ 5 u5587 dyno 4096 Feb 20 17:08 .. -rw------- 1 u5587 dyno 520 Feb 20 16:16 server.py -rwx------ 1 u5587 dyno 16176 Feb 20 16:16 test-print ~/backend $ id uid=13747(u13747) gid=13747(dyno) groups=13747(dyno) 

Answer: There are different user names u5587  vs u13747 in the output of ls and id. ~/backend $ ls -laq total 28 drwx------ 2 u5587 dyno 4096 Feb 20 16:16 . drwx------ 5 u5587 dyno 4096 Feb 20 17:08 .. -rw------- 1 u5587 dyno 520 Feb 20 16:16 server.py -rwx------ 1 u5587 dyno 16176 Feb 20 16:16 test-print ~/backend $ id uid=13747(u13747) gid=13747(dyno) groups=13747(dyno)  User u13747 does not have the permission to list the contents of the current directory or to access anything in this directory because it is owned by a different user and has no permissions for the group dyno or others. (This does not explain why cat test-print would work.)
Question: Similar to why use True is slower than use 1 in Python3 but I'm using pypy3 and not using the sum function. def sieve_num(n): nums = [0] * n for i in range(2, n): if i * i >= n: break if nums[i] == 0: for j in range(i*i, n, i): nums[j] = 1 return [i for i in range(2, n) if nums[i] == 0] def sieve_bool(n): nums = [False] * n for i in range(2, n): if i * i >= n: break if nums[i] == False: for j in range(i*i, n, i): nums[j] = True return [i for i in range(2, n) if nums[i] == False]  sieve_num(10**8) takes 2.55 s, but sieve_bool(10**8) takes 4.45 s, which is a noticeable difference. My suspicion was that [0]*n is somehow smaller than [False]*n and fits into cache better, but sys.getsizeof and vmprof line profiling are unsupported for PyPy. The only info I could get is that <listcomp> for sieve_num took 116 ms (19% of total execution time) while <listcomp> for sieve_bool tool 450 ms (40% of total execution time). Using PyPy 7.3.1 implementing Python 3.6.9 on Intel i7-7700HQ with 24 GB RAM on Ubuntu 20.04. With Python 3.8.10 sieve_bool is only slightly slower.

Answer: The reason is that PyPy uses a special implementation for "list of ints that fit in 64 bits". It has got a few other special cases, like "list of floats", "list of strings that contain only ascii", etc. The goal is primarily to save memory: a list of 64-bit integers is stored just like an array.array('l') and not a list of pointers to actual integer objects. You save memory not in the size of the list itself---which doesn't change---but in the fact that you don't need a very large number of small additional integer objects all existing at once. There is no special case for "list of boolean", because there are only ever two boolean objects in the first place. So there would be no memory-saving benefit in using a strategy like "list of 64-bit ints" in this case. Of course, we could do better and store that list with only one bit per entry, but it is not a really common pattern in Python; we just never got around to implementing that. So why it is slower, anyway? The reason is that in the "list of general objects" case, the JIT compiler needs to produce extra code to check the type of objects every time it reads an item from the list, and extra GC logic every time it puts an item into the list. This is not a lot of code, but in your case, I guess it doubles the length of the (extremely short) generated assembly for the inner loop doing nums[j] = 1. Right now, both in PyPy and CPython(*), the fastest is probably to use array.array('B') instead of a list, which both avoids that PyPy-specific issue and also uses substantially less memory (always a performance win if your data structures contain 10**8 elements). EDIT: (*) no, turns out that CPython is probably too slow for the memory bandwidth to be a limit. On my machine, PyPy is maybe 30-35% faster when using bytes. See also comments for a hack that speeds up CPython from 9x to 3x slower than PyPy, but which as usual slows down PyPy.
Question: My actual problem is in a higher dimension, but I am posting it in a smaller dimension to make it easy to visualize. I have a tensor of shape (2,3,4): x = torch.randn(2, 3, 4) tensor([[[-0.9118, 1.4676, -0.4684, -0.6343], [ 1.5649, 1.0218, -1.3703, 1.8961], [ 0.8652, 0.2491, -0.2556, 0.1311]], [[ 0.5289, -1.2723, 2.3865, 0.0222], [-1.5528, -0.4638, -0.6954, 0.1661], [-1.8151, -0.4634, 1.6490, 0.6957]]])  From this tensor, I need to select rows given by a list of indices along axis-1. Example, indices = torch.tensor([0, 2])  Expected Output: tensor([[[-0.9118, 1.4676, -0.4684, -0.6343]], [[-1.8151, -0.4634, 1.6490, 0.6957]]])  Output Shape: (2,1,4) Explanation: Select 0th row from x[0], select 2nd row from x[1]. (Came from indices) I tried using index_select like this: torch.index_select(x, 1, indices)  But the problem is that it is selecting the 0th and 2nd row for each item in x. It looks like it needs some modification I could not figure it out at the moment.

Answer: In your case, this is quite straightforward. An easy way to navigate through two dimensions in parallel is to use a range on the first axis and your indexing tensor on the second: >>> x[range(len(indices)), indices] tensor([[-0.9118, 1.4676, -0.4684, -0.6343], [-1.8151, -0.4634, 1.6490, 0.6957]])   In more general cases though, this would require the use of torch.gather:  First expand indices such that it has enough dimensions: index = indices[:,None,None].expand(x.size(0), -1, x.size(-1))   Then you can apply the function on x and index and squeeze dim=1: >>> x.gather(dim=-2, index=index)[:,0] tensor([[-0.9118, 1.4676, -0.4684, -0.6343], [-1.8151, -0.4634, 1.6490, 0.6957]])