eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Will Path('.').glob('*.ext') produce consistent ordering of results (assuming the files being globbed don't change)? It seems the glob ordering is based on the file system order (at least, for the old glob package). Will pathlib's glob order be changed by adding files to the directory (which will not be included in the glob)? Will this order be changed by the file system even if nothing is added to the specific directory (e.g., when other large file changes are made elsewhere on the system)? Over the course of several days? Or will the ordering remain consistent in all these cases? Just to clarify, I can't simple convert to a list and sort as there are too many file paths to fit into memory simultaneously. I'm hoping to achieve the same order each time as I will be doing some ML training, and want to set aside every nth file as validation data. This training will take several days, which is why I'm interested to know if the order remains stable over long times on the file system.

Answer: Checking the source code for the pathlib module, by chance, the latest commit points us directly to the relevant place:  Use os.scandir() as context manager in Path.glob().  So under the hood Path.glob uses os.scandir to get the directory entries. The docs of this function report that the results are unordered:  Return an iterator of os.DirEntry objects corresponding to the entries in the directory given by path. The entries are yielded in arbitrary order, and the special entries '.' and '..' are not included.  (emphasis mine)
Question: My debugging is setup in VS code, I can hit break points in the file I'm running via the launch.json config, but I can't get breakpoints in packages that are installed with PIP. How do I get breakpoints in these package files?

Answer: If you know where you python installation is on your computer do this:  Know where you python packets were installed. File -> add folder to workspace Add the breakpoints where necessary.  As an alternative i would advise to create a virtual environment and do the same thing but with the safety of virtual environment. Hope that helps.
Question: I'm trying to run this first code snippet provided by the Telethon documentation. But, after multiple problems (here and here), I ended up with this modified version: import os import sys from telethon.sync import TelegramClient, events # import nest_asyncio # nest_asyncio.apply() session_name = "<session_name>" api_id = <api_id> api_hash = "<api_hash>" os.chdir(sys.path[0]) if f"{session_name}.session" in os.listdir(): os.remove(f"{session_name}.session") async with TelegramClient(session_name, api_id, api_hash) as client: client.send_message('me', 'Hello, myself!') print(client.download_profile_photo('me')) @client.on(events.NewMessage(pattern='(?i).*Hello')) async def handler(event): await event.reply('Hey!') client.run_until_disconnected()  However now I'm getting these warnings:  usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:23: RuntimeWarning: coroutine 'MessageMethods.send_message' was never awaited RuntimeWarning: Enable tracemalloc to get the object allocation traceback /usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:24: RuntimeWarning: coroutine 'DownloadMethods.download_profile_photo' was never awaited RuntimeWarning: Enable tracemalloc to get the object allocation traceback /usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:30: RuntimeWarning: coroutine 'UpdateMethods._run_until_disconnected' was never awaited RuntimeWarning: Enable tracemalloc to get the object allocation traceback  when running the code on Jupyter. Now here are my questions:  what those warning messages mean and how should I address them? what is the expected result of this code if working properly? Should I receive a message in Telegram or something? Because I don't recive any messages other than the signin code.  What does the @ symbol at the beginning of the @client.on... line mean? what does that line is supposed to do? From this line onwards I do not understand the code. Would appreciate if you could help me understand it. 

Answer: Just add await the client.send_message('me', 'Hello, myself!') to solve that error and print afterdownload_profile_photo has done its work downloads an image to localhost so that may be why you don't see anything. You should read telethon documentation thoroughly and also how to use photo downloads correctly All the calls to the client have a delay and should always be awaited so that your code doesn't get blocked. You should read the asyncio tutorial The correct code would be: async with TelegramClient(session_name, api_id, api_hash) as client: await client.send_message('me', 'Hello, myself!') print(await client.download_profile_photo('me')) @client.on(events.NewMessage(pattern='(?i).*Hello')) async def handler(event): await event.reply('Hey!') #await client.run_until_disconnected()  The @ is a decorator and you should read the PEP related to decorators, but in short words, they execute a function before yours. In this case @client.on(events.NewMessage means:  When there is a new event that happens to be a message that matches the pattern specified handle it with this function called handler
Question: Say I have the following dataframe (a column of integers and a column with a list of integers)...  ID Found_IDs 0 12345 [15443, 15533, 3433] 1 15533 [2234, 16608, 12002, 7654] 2 6789 [43322, 876544, 36789]  And also a separate list of IDs... bad_ids = [15533, 876544, 36789, 11111]  Given that, and ignoring the df['ID'] column and any index, I want to see if any of the IDs in the bad_ids list are mentioned in the df['Found_IDs'] column. The code I have so far is: df['bad_id'] = [c in l for c, l in zip(bad_ids, df['Found_IDs'])]  This works but only if the bad_ids list is longer than the dataframe and for the real dataset the bad_ids list is going to be a lot shorter than the dataframe. If I set the bad_ids list to only two elements... bad_ids = [15533, 876544]  I get a very popular error (I have read many questions with the same error)... ValueError: Length of values does not match length of index  I have tried converting the list to a series (no change in the error). I have also tried adding the new column and setting all values to False before doing the comprehension line (again no change in the error). Two questions:  How do I get my code (below) to work for a list that is shorter than a dataframe?  How would I get the code to write the actual ID found back to the df['bad_id'] column (more useful than True/False)?  Expected output for bad_ids = [15533, 876544]:  ID Found_IDs bad_id 0 12345 [15443, 15533, 3433] True 1 15533 [2234, 16608, 12002, 7654] False 2 6789 [43322, 876544, 36789] True  Ideal output for bad_ids = [15533, 876544] (ID(s) are written to a new column or columns):  ID Found_IDs bad_id 0 12345 [15443, 15533, 3433] 15533 1 15533 [2234, 16608, 12002, 7654] False 2 6789 [43322, 876544, 36789] 876544  Code: import pandas as pd result_list = [[12345,[15443,15533,3433]], [15533,[2234,16608,12002,7654]], [6789,[43322,876544,36789]]] df = pd.DataFrame(result_list,columns=['ID','Found_IDs']) # works if list has four elements # bad_ids = [15533, 876544, 36789, 11111] # fails if list has two elements (less elements than the dataframe) # ValueError: Length of values does not match length of index bad_ids = [15533, 876544] # coverting to Series doesn't change things # bad_ids = pd.Series(bad_ids) # print(type(bad_ids)) # setting up a new column of false values doesn't change things # df['bad_id'] = False print(df) df['bad_id'] = [c in l for c, l in zip(bad_ids, df['Found_IDs'])] print(bad_ids) print(df) 

Answer: Using np.intersect1d to get the intersect of the two lists: df['bad_id'] = df['Found_IDs'].apply(lambda x: np.intersect1d(x, bad_ids)) ID Found_IDs bad_id 0 12345 [15443, 15533, 3433] [15533] 1 15533 [2234, 16608, 12002, 7654] [] 2 6789 [43322, 876544, 36789] [876544]  Or with just vanilla python using intersect of sets: bad_ids_set = set(bad_ids) df['Found_IDs'].apply(lambda x: list(set(x) & bad_ids_set))
Question: This is a question from Codewars: Complete the method/function so that it converts dash/underscore delimited words into camel casing. The first word within the output should be capitalized only if the original word was capitalized (known as Upper Camel Case, also often referred to as Pascal case). The input test cases are as follows: test.describe("Testing function to_camel_case") test.it("Basic tests") test.assert_equals(to_camel_case(''), '', "An empty string was provided but not returned") test.assert_equals(to_camel_case("the_stealth_warrior"), "theStealthWarrior", "to_camel_case('the_stealth_warrior') did not return correct value") test.assert_equals(to_camel_case("The-Stealth-Warrior"), "TheStealthWarrior", "to_camel_case('The-Stealth-Warrior') did not return correct value") test.assert_equals(to_camel_case("A-B-C"), "ABC", "to_camel_case('A-B-C') did not return correct value")  This is what I've tried so far: def to_camel_case(text): str=text str=str.replace(' ','') for i in str: if ( str[i] == '-'): str[i]=str.replace('-','') str[i+1].toUpperCase() elif ( str[i] == '_'): str[i]=str.replace('-','') str[i+1].toUpperCase() return str  It passes the first two tests but not the main ones :  test.assert_equals(to_camel_case("the_stealth_warrior"), "theStealthWarrior", "to_camel_case('the_stealth_warrior') did not return correct value") test.assert_equals(to_camel_case("The-Stealth-Warrior"), "TheStealthWarrior", "to_camel_case('The-Stealth-Warrior') did not return correct value") test.assert_equals(to_camel_case("A-B-C"), "ABC", "to_camel_case('A-B-C') did not return correct value")  What am I doing wrong? 

Answer: You may have a working implementation with slight errors as mentioned in your comments, but I propose that you:  split by the delimiters apply a capitalization for all but the first of the tokens rejoin the tokens  My implementation is: def to_camel_case(text): s = text.replace("-", " ").replace("_", " ") s = s.split() if len(text) == 0: return text return s[0] + ''.join(i.capitalize() for i in s[1:])  IMO it makes a bit more sense. The output from running tests is: >>> to_camel_case("the_stealth_warrior") 'theStealthWarrior' >>> to_camel_case("The-Stealth-Warrior") 'TheStealthWarrior' >>> to_camel_case("A-B-C") 'ABC'
Question: I have 2 tensors with .size of torch.Size([2272, 161]). I want to get mean-squared-error between them. However, I want it along each of the 161 channels, so that my error tensor has a .size of torch.Size([161]). How can I accomplish this? It seems that torch.nn.MSELoss doesn't let me specify a dimension.

Answer: For the nn.MSELoss you can specify the option reduction='none'. This then gives you back the squared error for each entry position of both of your tensors. Then you can apply torch.sum/torch.mean. a = torch.randn(2272,161) b = torch.randn(2272,161) loss = nn.MSELoss(reduction='none') loss_result = torch.sum(loss(a,b),dim=0)  I don't think there is a direct way to specify at the initialisation of the loss to which dimension to apply mean/sum. Hope that helps!
Question: Can a shape such as a rectangle have a smooth color gradient in Plotly? I define the shape with a solid fill color as: shapes=[dict( type='rect', xref='x', yref='paper', x0=box_from, x1=box_to, y0=0, y1=1, fillcolor='Green', opacity=0.07, layer='below', line=dict(width=0), )]  But I'd like the box not to have a solid color fill, but to have a smooth color gradient.  This is the example in the documentation I'm following: https://plotly.com/python/shapes/#highlighting-time-series-regions-with-rectangle-shapes The docs on fillcolor aren't very extensive: https://plotly.com/python/reference/#layout-shapes-items-shape-fillcolor I guess colorscales don't apply to shapes: https://plotly.com/python/builtin-colorscales/  My guess is the answer is a simple "not supported", but perhaps someone else knows better.

Answer: Someone will correct me if I'm wrong but I think that no, there is no straight implementation to fill with a gradient a shape. But to achieve a similar results you could plot several lines inside the rectangle specifying decreasing rgb values. For example I added this for loop after the first rectangle definition in the documentation code (changing also the rectangle fillcolor to white). for i in range(100): fig.add_shape(type='line', xref="x", yref="y", x0=2.5, x1=3.5, y0=i*(2/100), y1=i*(2/100), line=dict( color='rgb({}, {}, {})'.format((i/100*255),(i/100*255),(i/100*255)), width=3, ))  And the result is:  I know it's impractical and that it can increase a bit the running time but if you care only about the aesthetic it does the job.
Question: I'm trying to use mypy in my projects, but many of the instance attributes I use are only initialized after __init__, and not inside it. However, I do want to keep the good practice of declaring all instance attributes at __init__, so I need some complicated solutions to make this work. An example to how I want this to behave (currently mypy is complaining): from typing import Optional class Foo: def __init__(self, x: int): self.x = x self.y: int = None # will initialize later, but I know it will be an int def fill_values(self): self.y = self.x**2 def do(self) -> int: return self.x + self.y  Currently mypy complains about the assignment of self.y, and wants it to be Optional or None. If I agree with it and change the line to self.y: Optional[int] = None, then mypy complains on the return value of do, because self.y might be None. The only way I found around it is to add as assert before using self.y, like: assert self.y is not None, which mypy picks up and understands. However, starting each method with many asserts is quite hard. I have many such values, and usually one method that initializes all of them, and all other methods runs after it. I understand that mypy is rightfully complaining (the method do can be called before fill_values), but even when I try to prevent it I can't get mypy to accept this. I can extend this example by adding more functionality but mypy can't infer this: from typing import Optional class Foo: def __init__(self, x: int): self.x = x self.y: int = None # will initialize later, but I know it will be an int def fill_values(self): self.y = x**2 def check_values(self): assert self.y is not None def do(self) -> int: if self.y is None: self.fill_values() self.check_values() return self.x + self.y  Any idea of a more elegant solution that multiple assert statements and Optional types which obscure the code?

Answer: I have found this to work for me: class Foo: def __init__(self, x: int): self.x = x self.y: int # Give self.y a type but no value def fill_values(self): self.y = self.x ** 2 def do(self) -> int: return self.x + self.y  Essentially all you are doing is telling mypy that self.y will be an integer when (and if) it is initialised. Trying to call self.y before it is initialised will raise an error and you can check if it has been initialised using hasattr(self, "y"). Run it on mypy playground.
Question: I can globally replace a regular expression with re.sub(), and I can count matches with for match in re.finditer(): count++  Is there a way to combine these two, so that I can count my substitutions without making two passes through the source string? Note: I'm not interested in whether the substitution matched, I'm interested in the exact count of matches in the same call, avoiding one call to count and one call to substitute.

Answer: You can use re.subn. re.subn(pattern, repl, string, count=0, flags=0)  it returns (new_string, number_of_subs_made) For example purposes, I'm using the same example as @Shubham Sharma used. text = "Jack 10, Lana 11, Tom 12, Arthur, Mark" out_str, count = re.subn(r"(\d+)", repl='repl', string=text) # out_str--> 'Jack repl, Lana repl, Tom repl, Arthur, Mark' # count---> 3
Question: Someone's just asked me how to list all the virtual environments created with venv. I could only think of searching for pyvenv.cfg files to find them. Something like: from pathlib import Path venv_list = [str(p.parent) for p in Path.home().rglob('pyvenv.cfg')]  This could potentially include some false positives. Is there a better way to list all the virtual environment created with venv? NB: The question is about venv specifically, NOT Anaconda, virtualenv, etc.

Answer: On Linux/macOS this should get most of it find ~ -d -name "site-packages" 2>/dev/null Looking for directories under your home that are named "site-packages" which is where venv puts its pip-installed stuff. the /dev/null bit cuts down on the chattiness of things you don't have permission to look into. Or you can look at the specifics of a particular expected file. For example, activate has nondestructive as content. Then you need to look for a pattern than matches venv but not anaconda and the rest. find ~ -type f -name "activate" -exec egrep -l nondestructive /dev/null {} \; 2>/dev/null macos mdfind On macos, this is is pretty fast, using mdfind (locate on Linux would probably have similar performance. mdfind -name activate | egrep /bin/activate$| xargs -o egrep -l nondestructive 2>/dev/null | xargs -L 1 dirname | xargs -L 1 dirname  So we :  look for all activate files egrep to match only bin/activate files (mdfind matches on things like .../bin/ec2-activate-license) look for that nondestructive and print filename where there is a match. the 2 xargs -L 1 dirname allow us to "climb up" from /bin/activate to the virtual env's root.  Helper function with -v flag to show details. jvenvfindall(){ # search for Python virtual envs. -v for verbose details unset verbose OPTIND=1 while getopts 'v' OPTION; do case "$OPTION" in v) verbose=1 ;; ?) ;; esac done shift "$(($OPTIND -1))" local bup=$PWD for dn in $(mdfind -name activate | egrep /bin/activate$| xargs -o egrep -l nondestructive 2>/dev/null | xargs -L 1 dirname | xargs -L 1 dirname) do if [[ -z "$verbose" ]]; then printf "$dn\n" else printf "\n\nvenv info for $dn:\n" cd $dn echo space usage, $(du -d 0 -h) #requires the jq and jc utilities... to extract create and modification times echo create, mod dttm: $(stat . | jc --stat | jq '.[]|{birth_time, change_time}') tree -d -L 1 lib fi done cd $bup }  output: ... venv info for /Users/me/kds2/issues2/067.pip-stripper/010.fixed.p1.check_venv/venvtest: space usage, 12M . create, mod dttm: { "birth_time": "Apr 16 13:04:43 2019", "change_time": "Sep 30 00:00:39 2019" } lib └── python3.6 ...  Hmmm, disk usage is not that bad, but something similar for node_modules might save some real space.
Question: My current understanding of the quotechar parameter is that it surrounds the fields that are separated by a comma. I'm reading the csv documentation for python and have written a similar code to theirs as such:  import csv with open("test.csv", newline="") as file: reader = csv.reader(file, delimiter=",", quotechar="|") for row in reader: print(row)  My csv file contains the following: |Hello|,|My|,|name|,|is|,|John|  The output gives a list of strings as expected: ['Hello', 'My', 'name', 'is', 'John']  The problem arises when I have whitespace in between the commas in my csv file. For example, if i have a whitespace after the closing | of a field like such: |Hello| ,|My| ,|name| ,|is| ,|John|  It gives the same output as before but now there's a whitespace included in the strings in the list: ['Hello ', 'My ', 'name ', 'is ', 'John']  It was my understanding that the quotechar parameter would only consider what was between the | symbol. Any help is greatly appreciated!

Answer: The quotechar argument is  A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters. It defaults to '"'.  For example, If your csv file contains data of the form |Hello|,|My|,|name|,|is|,|"John"| |Hello|,|My|,|name|,|is|,|"Tom"|  then in that case you can't use the default quotechar which is " because its already present in entities of the csv data so to instruct the csv reader that you want "John" to be included as it is in the output you would specify the some other quotechar, it may be | or ; or any character depending on the requirements. The output now include John and Tom in quotation marks, ['Hello', 'My', 'name', 'is', '"John"'] ['Hello', 'My', 'name', 'is', '"Tom"']  Consider another example where csv field itself contains delimiter, consider the csv file contains "Fruit","Quantity","Cost" "Strawberry","1000","$2,200" "Apple","500","$1,100"  Now in such case you have to specify the quotechar explicitly to instruct the csv reader so that it can distinguish between actual delimiter (control character) and comma (literal characters) in the csv field. Now in this case the quotechar " will also work.  Now coming to your code, you have to replace the extra white space before the delimiter in the csv file with the empty string. You can do this in the following way: Try this: from io import StringIO with open("test.csv", newline="") as f: file = StringIO(f.read().replace(" ,", ",")) reader = csv.reader(file, delimiter=",", quotechar="|") for row in reader: print(row)  This outputs, ['Hello', 'My', 'name', 'is', 'John']
Question: I have a query I'm using: people = Person.objects.all().annotate(num_pets=Count('pets')) for p in people: print(p.name, p.num_pets == 0)  (Pet is ManyToOne with Person) But i'm actually not interested in the number of pets, but only on whether a person has any pets or not. How can this be done?

Answer: You can make use of an Exists expression [Django-doc] to determine if there exists a Pet for that Person. For example: from django.db.models import Exists, OuterRef Person.objects.annotate( has_pet=Exists(Pet.objects.filter(person=OuterRef('pk'))) ) Here the model is thus Pet that has a ForeignKey named person to Person. If the fields are named differently, then you should of course update the query accordingly.
Question: I am trying to copy some data from S3 bucket to redshift table by using the COPY command. The format of the file is PARQUET. When I run the execute the COPY command query, I get InternalError_: Spectrum Scan Error.   This is the first time I tried copying from a parquet file. Please help me if there is a solution for this. I am using boto3 in python. 

Answer: This generally happens for below reasons:  If there is a mismatch in number of columns between table and file. If the Column type of your file schema is incompatible with your target table column type.  Try going into the error logs. You might find partial log in cloud watch. From the screen shot you have uplaoded, you can also find a query number you have run. Got to aws redshift query editor and run below query to get the full log: select message from svl_s3log where query = '<<your query number>>' order by query,segment,slice;  Hope this helps !
Question: I'm trying to do maxpooling over channel dimension: class ChannelPool(nn.Module): def forward(self, input): return torch.max(input, dim=1)  but I get the error AttributeError: 'torch.return_types.max' object has no attribute 'dim' 

Answer: The torch.max function called with dim returns a tuple so: class ChannelPool(nn.Module): def forward(self, input): input_max, max_indices = torch.max(input, dim=1) return input_max  From the documentation of torch.max:  Returns a namedtuple (values, indices) where values is the maximum value of each row of the input tensor in the given dimension dim. And indices is the index location of each maximum value found (argmax).