eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In Python 2, there were two methods called blind/unblind. Their docs recommend that you should now use pkcs1_15, however they only show how to sign/verify a message. Their sample code looks like this: from Crypto.Signature import pkcs1_15 from Crypto.Hash import SHA256 from Crypto.PublicKey import RSA # Generate a new RSA key pair private_key = RSA.generate(3072) public_key = private_key.publickey() # Message which is larger than the modulus message = 'Some arbitrary text to blind.' message = message.encode('utf-8') hashed_message = SHA256.new(message) # Sign message on the senders side signed_message = pkcs1_15.new(private_key).sign(hashed_message) # On the receivers' side, you can verify the signed message with the public key pkcs1_15.new(public_key).verify(hashed_message, signed_message) # No exceptions raised, so we can conclude that the signature is valid.  Does anyone know what the code would look like, if one were to blind it instead?

Answer: Firstly: PyCrypto is no longer maintained, and one should use its mostly-API-compatible successor, PyCryptodome, instead. If one is willing to learn a new API, they even advocate to switch to the Cryptography library instead. According to their documentation, they fully dropped support for manually blinding and unblinding messages. Instead they ask you to specifically pick one of the supported padding schemes, as is done in the sample code in your question. Now of course, if you are looking for actual blind signatures as per your tag, then it seems that you'll have to move away from PyCrypto(dome), as they no longer support that. In fact there seems to be no out-of-the-box support in any of the big Python cryptographic libraries, based on a cursory search.
Question: I have tried a lot. Banning words doesn't help, removing certain characters doesn't help. The datetime module doesn't have a directive for this. It has things like %d which will give you today's day, for example 24. I have a date in the format of 'Tuesday 24th January' but I need it to be 'Tuesday 24 January'. Is there a way to remove st,nd,rd,th. Or is there an even better way? EDIT: even removing rd would remove it from Saturday. So that doesn't work either.

Answer: You can use a regex: import re d = 'Tuesday 24th January' d = re.sub(r'(\d+)(st|nd|rd|th)', r'\1', d) # \1 to restore the captured day print(d) # Output Tuesday 24 January  For Saturday 21st January: d = 'Saturday 21st January' d = re.sub(r'(\d+)(st|nd|rd|th)', r'\1', d) print(d) # Output Saturday 21 January
Question: I want to create a DataFrame in a fixture using the following code: @pytest.fixture def my_fun(spark_session): return spark_session.createDataFrame( [ (*['test', 'testy']) ], T.StructType([ T.StructField('mytest', T.StringType()), T.StructField('mytest2', T.StringType() ]) ) def test_something(my_fun): return  However, this fails with the following error: TypeError: StructType can not accept object 'test' in type <class 'str'>  If I use ('test', 'testy') instead of (*['test', 'testy']), it works. But shouldn't this be synonymous? (I'm using Python 3.8.13, pytest-7.0.1)

Answer: They are not the same. The round brackets in your example are not a tuple, they are just round brackets around a list. To make it a tuple you need to add a comma test = (*['test1', 'test2'],) # ('test1', 'test2')
Question: I hope you're doing okay. I've been trying to think how to solve the next problem, but I can't find a way to do it. Can you guys give me a hand, please? I have a dataframe with 4 columns, I want to add the remaining rows per group to have 3 Calendar Weeks, I want the new rows to keep the same value of ID of the group and display a NaN value for the Price and Attribute columns. import pandas as pd import numpy as np input = {'ID':['ITEM1', 'ITEM2', 'ITEM1', 'ITEM4', 'ITEM2', 'ITEM3', 'ITEM4', 'ITEM4'], 'Price':['11', '12', '11', '14', '12', '13', '14', '14' ], 'Attribute': ['A', 'B', 'A', 'D', 'B', 'C', 'D', 'D' ], 'Calendar Week':['1', '2', '2', '1', '3', '1', '3', '2'] } df = pd.DataFrame(input) df = df.sort_values(['ID', 'Calendar Week'], ascending = True).reset_index().drop(columns = 'index')  df =     ID Price Attribute Calendar Week     ITEM1 11 A 1   ITEM1 11 A 2   ITEM2 12 B 2   ITEM2 12 B 3   ITEM3 13 C 1   ITEM4 14 D 1   ITEM4 14 D 2   ITEM4 14 D 3     Expected output:     ID Price Attribute Calendar Week     ITEM1 11 A 1   ITEM1 11 A 2   ITEM1 NaN NaN 3   ITEM2 NaN NaN 1   ITEM2 12 B 2   ITEM2 12 B 3   ITEM3 13 C 1   ITEM3 NaN NaN 2   ITEM3 NaN NaN 3   ITEM4 14 D 1   ITEM4 14 D 2   ITEM4 14 D 3    

Answer: (df.set_index(["ID", "Calendar Week"]) .reindex(pd.MultiIndex.from_product([df["ID"].unique(), ["1", "2", "3"]], names=["ID", "Calendar Week"])) .reset_index()[df.columns])   you can move ID & Calendar Week to index part then reindex with the every possibility of ID versus Calendar Week generated with a product then move them back to columns and restore the original column order  to get  ID Price Attribute Calendar Week 0 ITEM1 11 A 1 1 ITEM1 11 A 2 2 ITEM1 NaN NaN 3 3 ITEM2 NaN NaN 1 4 ITEM2 12 B 2 5 ITEM2 12 B 3 6 ITEM3 13 C 1 7 ITEM3 NaN NaN 2 8 ITEM3 NaN NaN 3 9 ITEM4 14 D 1 10 ITEM4 14 D 2 11 ITEM4 14 D 3
Question: so the problem is that pypi.org hase been filtered by iranian government(yes , i know it's ridiculous!). i tried to install some python modules from Github downloaded files: pip install moduleName but every module has it's own dependencies and try to connect to pipy.org to reach them. then there will be an error during installation. is there any solution? your help will be much appreciated.

Answer: I live in a country that also blocks services, mostly streaming platforms. In theory, the way behind this is the same whether to watch Netflix or download python and its dependencies. That is you'll probably need to use a VPN. As said by d-dutchveiws, there's tons of videos and resources on how to set up a VPN. If you do end up using a paid VPN service I would just like to add that I lived in the UAE for a while and I found that some VPN services were blocked by the country themselves. I know NordVPN did not work/was blocked by the UAE so I ended up finding expressVPN and that worked. In other words, I'd be sure not to commit to any payment plan/only use free trials because even the VPN services can be blocked. Hope I helped a bit!
Question: In Python3.11 it's suggested to use TaskGroup for spawning Tasks rather than using gather. Given Gather will also return the result of a co-routine, what's the best approach with TaskGroup. Currently I have async with TaskGroup() as tg: r1 = tg.create_task(foo()) r2 = tg.create_task(bar()) res = [r1.result(), r2.result()]  Is there a more concise approach that can be used to achieve the same result?

Answer: The task groups were implemented more as a cleaner way to handle task lifetimes and exception handling, enabled greatly by the new exceptions group rework. I think its a popular misconception right now, but the TaskGroup is not a drop in replacement for all of the gather use-cases. For cases where you do not care about the results (which seems to be the only example I am seeing in new documentation and tutorials) it feels much more terse. When the task group has completed, it's still required by the user to pull results out of the completed coruntines. If you need values immediately then you can write it as you have it as res = [r1.result(), r2.result()] after the TaskGroup completes. A more terse syntax might be to gather the results after the TaskGroup completes with res = await asyncio.gather(r1, r2) (this will release the execution of your function which may or may not be desirable). This may look redundant to use both TaskGroup and gather, but the TaskGroup is solving a different purpose than what is provided by gather alone, being that it allows for waiting for your tasks with strong safety guarantees, logic around cancellation for failures, and grouping of exceptions. It might be possible to extend the default TaskGroup class to make this pattern easier. Here's one such idea that can keep track of which tasks were issued in the task group and provides a helper to fish out the results: class GatheringTaskGroup(asyncio.TaskGroup): def __init__(self): super().__init__() self.__tasks = [] def create_task(self, coro, *, name=None, context=None): task = super().create_task(coro, name=name, context=context) self.__tasks.append(task) return task def results(self): return [task.result() for task in self.__tasks]  async def foo(): return 1 async def bar(): return 2 async with GatheringTaskGroup() as tg: task1 = tg.create_task(foo()) task2 = tg.create_task(bar()) print(tg.results())  [1, 2]
Question: How do I create a date range in Polars (Python API) with only the last days of the months? This is the code I have: pl.date_range(datetime(2022,5,5), datetime(2022,8,10), "1mo", name="dtrange")  The result is: '2022-05-05', '2022-06-05', '2022-07-05', '2022-08-05' I would like to get: '2022-05-31', '2022-06-30', '2022-07-31' I know this is possible with Pandas with: pd.date_range(start=datetime(2022,5,5), end=datetime(2022,8,10), freq='M') 

Answer: I think you'd need to create the range of days and filter: (pl.date_range(datetime(2022,5,5), datetime(2022,8,10), "1d", name="dtrange") .to_frame() .filter( pl.col("dtrange").dt.month() < (pl.col("dtrange") + pl.duration(days=1)).dt.month() ) )  shape: (3, 1) ┌─────────────────────┐ │ dtrange │ │ --- │ │ datetime[μs] │ ╞═════════════════════╡ │ 2022-05-31 00:00:00 │ ├─────────────────────┤ │ 2022-06-30 00:00:00 │ ├─────────────────────┤ │ 2022-07-31 00:00:00 │ └─────────────────────┘
Question: Whenever I try to type-hint a list of strings, such as tricks: list[str] = []  , I get TypeError: 'type' object is not subscriptable. I follow a course where they use the same code, but it works for them. So I guess the problem is one of these differences between my an the courses environments. I use:  vs code anaconda python 3.8.15 jupyter notebook  Can someone help me fix that? I used the same code in normal .py files and it sill doesn't work, so that is probably not it. The python version should also not be the problem as this is kind of basic. Anaconda should also not cause such Error messages. Leaves the difference between vscode and pycharm, which is also strange. Therefore I don't know what to try.

Answer: You're on an old Python version. list[str] is only valid starting in Python 3.9. Before that, you need to use typing.List: from typing import List tricks: List[str] = []  If you're taking a course that uses features introduced in Python 3.9, you should probably get a Python version that's at least 3.9, though.
Question: Relevant portion of my code looks something like this: @directory_router.get("/youtube-dl/{relative_path:path}", tags=["directory"]) def youtube_dl(relative_path, url, name=""): """ Download """ relative_path, _ = set_path(relative_path) logger.info(f"{DATA_PATH}{relative_path}") if name: name = f"{DATA_PATH}{relative_path}/{name}.%(ext)s" else: name = f"{DATA_PATH}{relative_path}/%(title)s.%(ext)s" ydl_opts = { "outtmpl": name, # "quiet": True "logger": logger, "progress_hooks": [yt_dlp_hook], # "force-overwrites": True } with yt.YoutubeDL(ydl_opts) as ydl: try: ydl.download([url]) except Exception as exp: logger.info(exp) return str(exp)  I am using this webhook/end point to allow an angular app to accept url/name input and download file to folder. I am able to logger.info .. etc. output the values of the yt_dlp_hook, something like this: def yt_dlp_hook(download): """ download Hook Args: download (_type_): _description_ """ global TMP_KEYS if download.keys() != TMP_KEYS: logger.info(f'Status: {download["status"]}') logger.info(f'Dict Keys: {download.keys()}') TMP_KEYS = download.keys() logger.info(download)  Is there a way to stream a string of relevant variables like ETA, download speed etc. etc. to the front end? Is there a better way to do this?

Answer: You could use a Queue object to communicate between the threads. So when you call youtube_dl pass in a Queue that you can add messages inside yt_dlp_hook (you'll need to use partial functions to construct it). You'll be best off using asyncio to run the download at the same time as updating the user something like: import asyncio from functools import partial import threading from youtube_dl import YoutubeDL from queue import LifoQueue, Empty def main(): # Set the url to download url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Get the current event loop loop = asyncio.get_event_loop() # Create a Last In First Out Queue to communicate between the threads queue = LifoQueue() # Create the future which will be marked as done once the file is downloaded coros = [youtube_dl(url, queue)] future = asyncio.gather(*coros) # Start a new thread to run the loop_in_thread function (with the positional arguments passed to it) t = threading.Thread(target=loop_in_thread, args=[loop, future]) t.start() # While the future isn't finished yet continue while not future.done(): try: # Get the latest status update from the que and print it data = queue.get_nowait() print(data) except Empty as e: print("no status updates available") finally: # Sleep between checking for updates asyncio.run(asyncio.sleep(0.1)) def loop_in_thread(loop, future): loop.run_until_complete(future) async def youtube_dl(url, queue, name="temp.mp4"): """ Download """ yt_dlp_hook_partial = partial(yt_dlp_hook, queue) ydl_opts = { "outtmpl": name, "progress_hooks": [yt_dlp_hook_partial], } with YoutubeDL(ydl_opts) as ydl: return ydl.download([url]) def yt_dlp_hook(queue: LifoQueue, download): """ download Hook Args: download (_type_): _description_ """ # Instead of logging the data just add the latest data to the queue queue.put(download) if __name__ == "__main__": main()
Question: How can I download a specific format without using options like "best video", using the format ID... example: 139, see the picture ❯ yt-dlp --list-formats https://www.youtube.com/watch?v=BaW_jenozKc [youtube] Extracting URL: https://www.youtube.com/watch?v=BaW_jenozKc [youtube] BaW_jenozKc: Downloading webpage [youtube] BaW_jenozKc: Downloading android player API JSON [info] Available formats for BaW_jenozKc: ID EXT RESOLUTION FPS CH │ FILESIZE TBR PROTO │ VCODEC VBR ACODEC ABR ASR MORE INFO ───────────────────────────────────────────────────────────────────────────────────────────────────────────── 139 m4a audio only 2 │ 58.59KiB 48k https │ audio only mp4a.40.5 48k 22k low, m4a_dash 249 webm audio only 2 │ 58.17KiB 48k https │ audio only opus 48k 48k low, webm_dash 250 webm audio only 2 │ 76.07KiB 63k https │ audio only opus 63k 48k low, webm_dash 140 m4a audio only 2 │ 154.06KiB 128k https │ audio only mp4a.40.2 128k 44k medium, m4a_dash 251 webm audio only 2 │ 138.96KiB 116k https │ audio only opus 116k 48k medium, webm_dash 17 3gp 176x144 12 1 │ 55.79KiB 45k https │ mp4v.20.3 45k mp4a.40.2 0k 22k 144p 160 mp4 256x144 15 │ 135.08KiB 113k https │ avc1.4d400c 113k video only 144p, mp4_dash 278 webm 256x144 30 │ 52.22KiB 44k https │ vp9 44k video only 144p, webm_dash 133 mp4 426x240 30 │ 294.27KiB 246k https │ avc1.4d4015 246k video only 240p, mp4_dash 242 webm 426x240 30 │ 33.27KiB 28k https │ vp9 28k video only 240p, webm_dash 134 mp4 640x360 30 │ 349.59KiB 292k https │ avc1.4d401e 292k video only 360p, mp4_dash 18 mp4 640x360 30 2 │ ~525.60KiB 420k https │ avc1.42001E 420k mp4a.40.2 0k 44k 360p 243 webm 640x360 30 │ 75.55KiB 63k https │ vp9 63k video only 360p, webm_dash 135 mp4 854x480 30 │ 849.41KiB 710k https │ avc1.4d401f 710k video only 480p, mp4_dash 244 webm 854x480 30 │ 165.49KiB 138k https │ vp9 138k video only 480p, webm_dash 22 mp4 1280x720 30 2 │ ~ 1.82MiB 1493k https │ avc1.64001F 1493k mp4a.40.2 0k 44k 720p 136 mp4 1280x720 30 │ 1.60MiB 1366k https │ avc1.4d401f 1366k video only 720p, mp4_dash 247 webm 1280x720 30 │ 504.68KiB 420k https │ vp9 420k video only 720p, webm_dash 137 mp4 1920x1080 30 │ 2.11MiB 1803k https │ avc1.640028 1803k video only 1080p, mp4_dash 248 webm 1920x1080 30 │ 965.31KiB 804k https │ vp9 804k video only 1080p, webm_dash  I tried using the format url, but it didn't work

Answer: yt-dlp -f <format_id> <url>
Question: Consider a Python protocol attribute which is also annotated with a protocol. I found in that case, both mypy and Pyright report an error even when my custom datatype follows the nested protocol. For example in the code below Outer follows the HasHasA protocol in that it has hasa: HasA because Inner follows HasA protocol. from dataclasses import dataclass from typing import Protocol class HasA(Protocol): a: int class HasHasA(Protocol): hasa: HasA @dataclass class Inner: a: int @dataclass class Outer: hasa: Inner def func(b: HasHasA): ... o = Outer(Inner(0)) func(o)  However, mypy shows the following error. nested_protocol.py:22: error: Argument 1 to "func" has incompatible type "Outer"; expected "HasHasA" [arg-type] nested_protocol.py:22: note: Following member(s) of "Outer" have conflicts: nested_protocol.py:22: note: hasa: expected "HasA", got "Inner"  What's wrong with my code?

Answer: There's an issue on GitHub which is almost exactly the same as your example. I think the motivating case on the mypy docs explains quite well why this is illegal. Bringing a structural analogy to your example, let's fill in an implementation for func and tweak Inner slightly: def func(b: HasHasA) -> None: b.hasa.a += 100 - 100 @dataclass class Inner: a: bool o = Outer(Inner(bool(0))) func(o) if o.hasa.a is False: print("Oh no! This is still False!") else: print("This is true now!")  This is of course a contrived example, but it shows that if the type-checker didn't warn you against this, the inner protocol can type-widen the inner type and perform value mutation, and you may silently perform type-unsafe operations. As suggested by the mypy documentation, the solution is to make the outer protocol's variable read-only: class HasHasA(Protocol): @property def hasa(self) -> HasA: ...
Question: I'm trying to call an async function containing an await function from the main thread without halting computation in the main thread. I've looked into similar questions employing a variety of solutions two of which I have demonstrated below, however, none of which seem to work with my current setup involving aioconsole and asyncio. Here is a simplified version of what I currently have implemented: import aioconsole import asyncio async def async_input(): line = await aioconsole.ainput(">>> ") print("You typed: " + line) if __name__ == "__main__": asyncio.run(async_input()) print("This text should print instantly. It should not wait for you to type something in the console.") while True: # Do stuff here pass   I have also tried replacing asyncio.run(async_input()) with the code below but that hangs the program even after entering console input. loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.create_task(async_input()) loop.run_forever()   What's currently happening:  Function is created ✓ Function is called from the main thread ✓ The program stops and awaits the completion of the async function. ✗ The processing that should be performed in parallel with the async function is performed sequentially. ✗  Output >>>  What should happen:  Function is created ✓ Function is called from the main thread ✓ The program continues while the async function runs in the background. ✓ The processing following is performed in parallel with the async function. ✓  Expected output >>> This text should print instantly. It should not wait for you to type something in the console.  Python version: 3.10.8  Operating System: MacOS Ventura

Answer: As pointed out to me by @flakes there were a couple of problems with my approach:  asyncio.run is a blocking function so wrapping my async function calls in it was preventing any following code from running. The main content needs to be wrapped in an asynchronous function to give the indented async function time to run. The intended async function runs while the main thread is waiting so downtime must be introduced in the main function. This can be easily done with await asyncio.sleep(x).  The introduction of all this fixes the problem while allowing the main content to run synchronously: import aioconsole import asyncio async def async_input(): line = await aioconsole.ainput(">>> ") print("You typed: " + line) async def main(): input_task = asyncio.create_task(async_input()) print("This text should print instantly. It should not wait for you to type something in the console.") while True: # Do stuff here await asyncio.sleep(0.1) await input_task if __name__ == "__main__": asyncio.run(main())
Question: Say I have df as follows: MyCol Red Motor Green Taxi Light blue small Taxi Light blue big Taxi  I would like to split the color and the vehicle into two columns. I used this command to split the last word. But sometimes, there is a 'big' or 'small' associated with the car name. How can do the splitting with conditions? df[['color','vehicle']] = df.myCol.str.rsplit(pat=' ', n=1, expand=True) 

Answer: I think the best approach is to use extract with a regex pattern df['MyCol'].str.extract('^(.*?)\s((?:small|big)?\s?\w+)$')    0 1 0 Red Motor 1 Green Taxi 2 Light blue small Taxi 3 Light blue big Taxi  Regex details:  ^: Matches start of the string (.*?): first capturing group  .*?: matches any character zero or more times but as few times as possible (lazy match)   \s: Matches the space ((?:small|big)?\s?\w+): Second capturing group  (?:small|big)? : matches small or big zero or one time \s?: matches space zero or one time \w+: matches word characters oner or more times   $: matches end of the string  The Series.str.extract is used here to extracts two groups using a regular expression. The first group is before a whitespace and the second group is after the whitespace. The second group may contain the word "small" or "big" and returns a new DataFrame with two columns containing the extracted groups.