eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: There appear to be several ways to determine if a set is a subset of another set, but I haven't been able to find something concise that determines whether all elements of a list (or array), including duplicate values, appear in another list (or array). For example, for the hypothetical function contains_all(A, B) which checks whether all elements of B are contained in A, these are some expected outputs: contains_all([11, 4, 11, 6], [6, 11]) returns True (order doesn't matter) contains_all([11, 4, 11, 6], [11, 9]) returns False (no 9) contains_all([11, 4, 11, 6], [11, 11]) returns True contains_all([11, 4, 11, 6], [11, 11, 11]) returns False (only two 11's) contains_all([11, 4, 11, 6], [6, 6]) returns False (only one 6) contains_all([11, 4, 11, 6], [11, 4, 6, 11]) returns True contains_all([11, 4, 11, 6], [11, 4, 11, 6, 5]) returns False (no 5) The fourth and fifth examples above, specifically, are what I'm having trouble implementing. set(B).issubset(A) or list comprehensions cover the other cases, but not these, since sets don't have duplicate elements. Is there a concise way to do this? If not, what would be the best way to approach writing a function that does this? It seems something like this may be possible with collections.Counter objects or multisets, but I'm not sure how to go about it.

Answer: You are correct, collections.Counter is a good way to go. You just need to go over the B counter and check if the the value is smaller or equal. Do it in all() to check all the key-value pairs def contains_all(a, b): counter_a = Counter(a) counter_b = Counter(b) return all(v <= counter_a[k] for k, v in counter_b.items())  Edit user2357112 answer is much nicer, but apply to Pyton3.10 or newer. For older versions you can use this answer.
Question: I have a pretrained pytorch model which is saved in .pth format. How can i use it for prediction on new dataset in a separate python file. I need a detailed guide.

Answer: To use a pretrained model you should load the state on a new instance of the architecture as explained in the docs/tutorials: Here models is imported beforehand: model = models.vgg16() model.load_state_dict(torch.load('model_weights.pth')) # This line uses .load() to read a .pth file and load the network weights on to the architecture. model.eval() # enabling the eval mode to test with new samples.  If you are using a custom architecture you only need to change the first line. model = MyCustomModel()  After enabling the eval mode, you can proceed as follows:  Load your data into a Dataset instance and then in a DataLoader. Make your predictions with the data. Calculate metrics on the results.  More about Dataset and DataLoader here.
Question: My understanding of how a ThreadPoolExecutor works is that when I call #submit, tasks are assigned to threads until all available threads are busy, at which point the executor puts the tasks in a queue awaiting a thread becoming available. The behavior I want is to block when there is not a thread available, to wait until one becomes available and then only submit my task. The background is that my tasks are coming from a queue, and I only want to pull messages off my queue when there are threads available to work on these messages. In an ideal world, I'd be able to provide an option to #submit to tell it to block if a thread is not available, rather than putting them in a queue. However, that option does not exist. So what I'm looking at is something like:  with concurrent.futures.ThreadPoolExecutor(max_workers=CONCURRENCY) as executor: while True: wait_for_available_thread(executor) message = pull_from_queue() executor.submit(do_work_for_message, message)  And I'm not sure of the cleanest implementation of wait_for_available_thread. Honestly, I'm surprised this isn't actually in concurrent.futures, as I would have thought the pattern of pulling from a queue and submitting to a thread pool executor would be relatively common.

Answer: One approach might be to keep track of your currently running threads via a set of Futures:  active_threads = set() def pop_future(future): active_threads.pop(future) with concurrent.futures.ThreadPoolExecutor(max_workers=CONCURRENCY) as executor: while True: while len(active_threads) >= CONCURRENCY: time.sleep(0.1) # or whatever message = pull_from_queue() future = executor.submit(do_work_for_message, message) active_threads.add(future) future.add_done_callback(pop_future)  A more sophisticated approach might be to have the done_callback be the thing that triggers a queue pull, rather than polling and blocking, but then you need to fall back to polling the queue if the workers manage to get ahead of it.
Question: I get the following error { "errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1051'", "errorType": "OSError", "stackTrace": [ " File \"/var/lang/lib/python3.8/imp.py\", line 234, in load_module\n return load_source(name, filename, file)\n", " File \"/var/lang/lib/python3.8/imp.py\", line 171, in load_source\n module = _load(spec)\n", " File \"<frozen importlib._bootstrap>\", line 702, in _load\n", " File \"<frozen importlib._bootstrap>\", line 671, in _load_unlocked\n", " File \"<frozen importlib._bootstrap_external>\", line 843, in exec_module\n", " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n", " File \"/var/task/app.py\", line 3, in <module>\n nltk.download('stopwords')\n", " File \"/var/task/nltk/downloader.py\", line 777, in download\n for msg in self.incr_download(info_or_id, download_dir, force):\n", " File \"/var/task/nltk/downloader.py\", line 642, in incr_download\n yield from self._download_package(info, download_dir, force)\n", " File \"/var/task/nltk/downloader.py\", line 699, in _download_package\n os.makedirs(download_dir)\n", " File \"/var/lang/lib/python3.8/os.py\", line 213, in makedirs\n makedirs(head, exist_ok=exist_ok)\n", " File \"/var/lang/lib/python3.8/os.py\", line 223, in makedirs\n mkdir(name, mode)\n" ] }  when testing my lambda function. I don't understand what this error is telling me to do about the docker image I am using, if that even is the correct route to explore. What should I do

Answer: AWS Lambda is not a generic docker runner. The docker containers you deploy to Lambda have to comply with the AWS Lambda runtime environment. The docker image you are using is trying to write to the path /home/sbx_user1051 apparently. On AWS Lambda the file system is always read-only except for the /tmp path. You will have to modify the code running in the docker image to prevent it from writing anywhere else but /tmp/.
Question: I'd like how to create and print associative arrays in python3... like in bash I do: declare -A array array["alfa",1]="text1" array["beta",1]="text2" array["alfa",2]="text3" array["beta",2]="text4"  In bash I can do echo "${array["beta",1]}" to access data to print "text2". How can I define a similar array in python3 and how to access to data in a similar way? I tried some approaches, but none worked. Stuff like this: array = () array[1].append({ 'alfa': "text1", 'beta': "text2", })  But I can't access to data with print(array['beta', 1]). It is not printing "text2" :(

Answer: It looks like you want a dictionary with compound keys: adict = { ("alfa", 1): "text1", ("beta", 1): "text2", ("alfa", 2): "text3", ("beta", 2): "text4" } print(adict[("beta", 1)])
Question: Based on this answer (among others) it seems like f-strings is [one of] the preferred ways to convert to hexadecimal representation. While one can specify an explicit target length, up to which to pad with leading zeroes, given a goal of an output with an even number of digits, and inputs with an arbitrary # of bits, I can imagine:  pre-processing to determine the number of bits of the input, to feed an input-specific value in to the fstring, or post-processing a-la out = "0"+f"{val:x}" if len(f"{val:x}") % 2 else f"{val:02x}" (or even using .zfill())  The latter seems like it might be more efficient than the former - is there a built-in way to do this with fstrings, or a better alternative? Examples of input + expected output: [0x]1 -> [0x]01 [0x]22 -> [0x]22 [0x]333 -> [0x]0333 [0x]4444 -> [0x]4444  and so on.

Answer: Here's a postprocessing alternative that uses assignment expressions (Python 3.8+): print((len(hx:=f"{val:x}") % 2) * '0' + hx)  If you still want a one-liner without assignment expressions you have to evaluate your f-string twice: print((len(f"{val:x}") % 2) * '0' + f"{val:x}")  As a two-liner hx = f"{val:x}" print((len(hx) % 2) * '0' + hx)  And one more version: print(f"{'0'[:len(hex(val))%2]}{val:x}")
Question: I am doing poetry lock  Then I open the poetry.lock file and see that the metadata.files block does not contain hashes: [metadata.files] aiohttp = [] aiosignal = [] apscheduler = [] ...  Before, it wasn't like that. What could be the reasons for empty hashes?

Answer: You are probably running into this issue https://github.com/python-poetry/poetry/issues/5970 Just upgrade to poetry 1.1.14 or the prereleases for the 1.2 series.
Question: I was doing leetcode when I had to do some arithmetic with rational numbers (both numerator and denominator integers). I needed to count slopes in a list. In python collections.Counter( [ x/y if y != 0 else "inf" for (x,y) in points ] )  did the job, and I passed all the tests with it. ((edit: they've pointed out in the comments that in that exercise numbers were much smaller, not general 32 bit integers)) I wonder if this is correct, that is, python correctly recognizes if a/b == c/d as rationals, for a,b,c,d 32 bit integers. I am also interested the case for c++, and any additional facts that may be useful (footguns, best practices, theory behind it if not too long etc). Also this question seems frequent and useful, but I don't really find anything about it (give me the duplicates!), maybe I am missing some important keywords?

Answer: It's not safe, and I've seen at least one LeetCode problem where you'd fail with that (maybe Max Points on a Line). Example: a = 94911150 b = 94911151 c = 94911151 d = 94911152 print(a/b == c/d) print(a/b) print(c/d)  Both a/b and c/d are the same float value even though the slopes actually differ (Try it online!): True 0.9999999894638303 0.9999999894638303  You could use fractions.Fraction(x, y) or the tuple (x//g, y//g) after g = math.gcd(d, y) ( if I remember correctly, this is more lightweight/efficient than the Fraction class).
Question: When trying to login into a Gmail account using SMTP, this error message occurs: SMTPAuthenticationError(code, resp) smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted.  Code causing the error: import smtplib server = smtplib.SMTP("smtp.gmail.com", 587) server.starttls() server.login("sending@gmail.com", "your_password") message = "TEST" server.sendmail("sending@gmail.com", "receiving@gmail.com", message) server.quit() 

Answer: Google has disabled the ability to enable less secure apps as of May 2022. Because of this, the previous solution of enabling less secure apps is no longer valid. Steps:  Go into your sending email address and make your way to the settings. Find two-step authentication and enable it. Under two-step authentication there should be a tab labeled App passwords. Click on it then select mail as the app and your device of choice Use the password generated from the app password as the password for your Gmail account.  Credits to: Link to source
Question: I have a dataframe that looks like this: dict = {'companyId': {0: 198236, 1: 198236, 2: 900814, 3: 153421, 4: 153421, 5: 337815}, 'region': {0: 'Europe', 1: 'Europe', 2: 'Asia-Pacific', 3: 'North America', 4: 'North America', 5:'Africa'}, 'value': {0: 560, 1: 771, 2: 964, 3: 217, 4: 433, 5: 680}, 'type': {0: 'actual', 1: 'forecast', 2: 'actual', 3: 'forecast', 4: 'actual', 5: 'forecast'}} df = pd.DataFrame(dict) companyId region value type 0 198236 Europe 560 actual 1 198236 Europe 771 forecast 2 900814 Asia-Pacific 964 actual 3 153421 North America 217 forecast 4 153421 North America 433 actual 5 337815 Africa 680 forecast  I can't seem to figure out a way to filter out certain rows based on the following condition: If there are two entries under the same companyId, as is the case for 198236 and 153421, I want to keep only the entry where type is actual. If there is only one entry under a companyId, as is the case for 337815 and 900814, I want to keep that row, irrespective of the value in column type. Does anyone have an idea how to go about this?

Answer: You can use a groupby and transform to create boolean indexing: #Your condition i.e. retain the rows which are not duplicated and those # which are duplicated but only type==actual. Lets express that as a lambda. to_filter = lambda x: (len(x) == 1) | ((len(x) > 1) & (x == 'actual')) #then create a boolean indexing mask as below m = df.groupby('companyId')['type'].transform(to_filter) #then filter your df with that m: df[m]: companyId region value type 0 198236 Europe 560 actual 2 900814 Asia-Pacific 964 actual 4 153421 North America 433 actual 5 337815 Africa 680 forecast
Question: I want the store geospatial information in Redis. I am executing the following code from redis import Redis redis_con = Redis(host="localhost", port=6379) redis_con.geoadd("Sicily", 13.361389, 38.115556, "Palermo")  But ı got error like that raise DataError("GEOADD allows either 'nx' or 'xx', not both") redis.exceptions.DataError: GEOADD allows either 'nx' or 'xx', not both 

Answer: This will work for you: from redis import Redis redis_con = Redis(host="localhost", port=6379) coords = (13.361389, 38.115556, "Palermo") redis_con.geoadd("Sicily", coords)  The signature for geoadd is: geoadd(name, values, nx=False, xx=False, ch=False) name: Union[bytes, str, memoryview] values: Sequence[Union[bytes, memoryview, str, int, float]] nx (bool, default: False) xx (bool, default: False) ch (bool, default: False)  You need to specify your coordinates as a sequence like a list or tuple, because right now you're specifying arguments so that the method thinks you've specified nx, xx, and ch.
Question: Poetry official documentation strictly recommends sticking with the official installer. However, homebrew has poetry formulae. brew install poetry  Usually, I like to keep everything I can in homebrew to manage installations easily. What is the drawback and risks of installing poetry using homebrew instead of the recommended installation script?

Answer: The drawback is that poetry will be unable to upgrade itself (I've no idea what'd actually happen), and you'll not be able to install specific poetry versions. Homebrew installed poetry will probably also use the Homebrew-installed Python environment, etc, instead of having its own isolated venv to execute from. If you use homebrew to install poetry, don't try to manage that installation any way outside of homebrew. Otherwise, it's probably fine. Other misc drawbacks include:  Can't control specific Python version used to run Poetry (want to get the latest core Python speedups!) Uncertain support and management of poetry plugins (part of "managing the installation") Can't run multiple poetry versions side-by-side  edit My personal recommendation is to install pipx using homebrew, and then install poetry via pipx (as Poetry themselves now also recommend): brew install pipx pipx ensurepath pipx install poetry # latest version pipx install poetry==1.2.2 --suffix=@1.2 pipx install poetry==1.3.2 --suffix=@1.3 pipx install poetry==1.6.1 --suffix=@1.6  so you'll get something like: ❯ pipx list venvs are in /Users/redacted/.local/pipx/venvs apps are exposed on your $PATH at /Users/redacted/Code/dotfiles/bin package hatch 1.7.0, installed using Python 3.11.5 - hatch package poetry 1.2.2, installed using Python 3.11.5 - poetry package poetry 1.3.2 (poetry@1.3), installed using Python 3.11.5 - poetry@1.3 package poetry 1.6.1 (poetry@1.6), installed using Python 3.11.5 - poetry@1.6 ❯ poetry --version Poetry (version 1.2.2) ❯ poetry@1.3 --version Poetry (version 1.3.2)  e.g. via: brew install pipx pipx ensurepath pipx install poetry # latest version pipx install poetry==1.2.2 --suffix=@1.2 pipx install poetry==1.3.2 --suffix=@1.3 pipx install poetry==1.6.1 --suffix=@1.6  Poetry is still under very active development, and its API is not very stable. This gives me full control of what version to use/when, e.g. across different projects.
Question: I am wondering if it's possible to install fonts to use in altair to use in alt.TitleParams. For this example, without font specified, I get a default font and size. import altair as alt import pandas as pd source = pd.DataFrame({ 'a': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'], 'b': [28, 55, 43, 91, 81, 53, 19, 87, 52] }) alt.Chart(source, title = alt.TitleParams(text = 'Example Chart')).mark_bar().encode( x='a', y='b' )   Changing the font size I get the bigger letters: alt.Chart(source, title = alt.TitleParams(text = 'Example Chart', fontSize=24)).mark_bar().encode( x='a', y='b' )   But, when I add a font style, the size doesn't work anymore: alt.Chart(source, title = alt.TitleParams(text = 'Example Chart' , fontSize=24 , fontStyle = 'Arial')).mark_bar().encode( x='a', y='b' )   And the text looks always the same regardless of what font is specified: alt.Chart(source, title = alt.TitleParams(text = 'Example Chart' , fontSize=24 , fontStyle = 'Calibri')).mark_bar().encode( x='a', y='b' )  Same thing:  I would like to know how I can display the correct font, not only with standard fonts, but with non-standard ones and how to install them.

Answer: fontStyle refers to the style of the font, such as "bold", "italic", etc. If you want to specify a font by name, use the font parameter: alt.Chart( source, title=alt.Title( # Since altair 5 you can use just Title instead of TitleParams text='Example Chart', fontSize=24, fontStyle='italic', font='Times' ) ).mark_bar().encode( x='a', y='b' )
Question: I'm trying to understand the value-added of using fastai's fastcore.basics.patch_to decorator. Here's the fastcore way: from fastcore.basics import patch_to class _T3(int): pass @patch_to(_T3) def func1(self, a): return self + a  And here's the simple monkey-patching approach: class simple_T3(int): pass def func1(self, a): return self + a simple_T3.func1 = func1  Inspecting the two classes does not reveal any differences. I understand that simple monkey-patching might cause problems in more complex cases, so it would be great to know what such cases are? In other words, what's the value-added of fastcore.basics.patch_to?

Answer: TL;DR More informative debugging messages, better IDE support.  Answer patch and patch_to are decorators in the fastcore basics module that are helpful to make the monkey_patched method to look more like as if it was a method originally placed inside the Class, the classical way (pun intended). If you create a function outside a class and then monkey-patch it, the outsider method typically has different attributes, such as its name, module, and documentation, compared to the original function. This can be confusing and unhelpful when debugging or working with the "outsider" function. Source: Official documentation: https://github.com/fastai/fastcore/blob/master/nbs/01_basics.ipynb  Usage suggestion Consider using patch instead of patch_to, because this way you can add type annotations. from fastcore.basics import patch class _T3(int): pass @patch def func1(self: _T3, a): return self + a   What if I don't want to use the library? Credits: Kai Lichtenberg fastcore itself is extremely low weight: The only external library used is numpy (and dataclasses if your python is < 3.7). But if you really want to not use it, here's an implementation with only two built-in dependencies: import functools from copy import copy from types import FunctionType def copy_func(f): "Copy a non-builtin function (NB `copy.copy` does not work for this)" if not isinstance(f,FunctionType): return copy(f) fn = FunctionType(f.__code__, f.__globals__, f.__name__, f.__defaults__, f.__closure__) fn.__dict__.update(f.__dict__) return fn def patch_to(cls, as_prop=False): "Decorator: add `f` to `cls`" if not isinstance(cls, (tuple,list)): cls=(cls,) def _inner(f): for c_ in cls: nf = copy_func(f) # `functools.update_wrapper` when passing patched function to `Pipeline`, so we do it manually for o in functools.WRAPPER_ASSIGNMENTS: setattr(nf, o, getattr(f,o)) nf.__qualname__ = f"{c_.__name__}.{f.__name__}" setattr(c_, f.__name__, property(nf) if as_prop else nf) return f return _inner def patch(f): "Decorator: add `f` to the first parameter's class (based on f's type annotations)" cls = next(iter(f.__annotations__.values())) return patch_to(cls)(f)  class MyClass(): def __init__(self): pass @patch def new_fun(self:MyClass): print("I'm a patched function!") MyInstance = MyClass() MyInstance.new_fun()  "I'm a patched function!"
Question: Is there a difference between ThreadPoolExecutor from concurrent.futures and ThreadPool from multiprocessing.dummy? This article suggests that ThreadPool queries the "threads" (task) to the different "threads" of the CPU. Does concurrent.futures do the same or will the "threads" (tasks) query to a single "thread" of a CPU?

Answer: The multiprocessing.dummy.ThreadPool is a copy of the multiprocessing.Pool API which uses threads rather than processes, which leads to some weirdness since thread and processes are very different, including returning a AsyncResult type which only it understands. The concurrent.futures.ThreadPoolExecutor is a subclass concurrent.futures.Executor which is a newer simpler API, developped with both processes and threads in mind and so returns a common concurrent.futures.Future. On a very broad and far look, both do the same but concurrent.futures.ThreadPoolExecutor does it better.  References: From the multiprocessing.dummy documentation:  multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module. In particular, the Pool function provided by multiprocessing.dummy returns an instance of ThreadPool, which is a subclass of Pool that supports all the same method calls but uses a pool of worker threads rather than worker processes.  From the multiprocessing.dummy.ThreadPool documentation  A ThreadPool shares the same interface as Pool, which is designed around a pool of processes and predates the introduction of the concurrent.futures module. As such, it inherits some operations that don’t make sense for a pool backed by threads, and it has its own type for representing the status of asynchronous jobs, AsyncResult, that is not understood by any other libraries. Users should generally prefer to use concurrent.futures.ThreadPoolExecutor, which has a simpler interface that was designed around threads from the start, and which returns concurrent.futures.Future instances that are compatible with many other libraries, including asyncio.
Question: How do I fix this error? It seems feedparser does not support mypy typings? I could not find a typeshed implementation for feedparser UPDATE 1 I see an option called ignore_missing_imports that I can add to pyproject.toml. Isn't it a bad idea to do this?

Answer:  I see an option called ignore_missing_imports that I can add to pyproject.toml. Isn't it a bad idea to do this?  Yes, it usually is a bad idea to enable this on all modules. Consider using a more constrained approach: You can ignore missing imports only for this specific package by adding a [tools.mypy.override] section in pyproject.toml. This way you don't need enable the flag on everything. [[tool.mypy.overrides]] module = "feedparser.*" ignore_missing_imports = true  The work for supporting typing in feedparser has been merged to the develop branch and you should be able to remove this workaround when it is released.