eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: so i have pasted my complete code for your reference, i want to know what's the use of ppf and cdf here? can you explain it? i did some research and found out that ppf(percent point function) is an inverse of CDF(comulative distribution function) if they really are, shouldn't this code work if i replaced ppf and cdf as 1/cdf and 1/ppf respectively? please explain this to me, the difference between the two. and how to and when to use which this is, btw, hypothesis testing. and sorry for so many comments, just a habit of explaining everything for my future reference.(do point me out if any of my comments is wrong regarding the same) ball_bearing_radius = [2.99, 2.99, 2.70, 2.92, 2.88, 2.92, 2.82, 2.83, 3.06, 2.85] import numpy as np from math import sqrt from scipy.stats import norm # h1 : u != U_0 # h0 : u = u_0 #case study : ball bearing example, claim is that radius = 3, do hypothesis testing mu_0 = 3 sigma = 0.1 #collect sample sample = ball_bearing_radius #compute mean mean = np.mean(sample) #compute n n = len(sample) #compute test statistic z = (mean - mu_0) /(sigma/sqrt(n)) #set alpha a = 0.01 #------------------------- #calculate the z_a/2, by using percent point function of the norm of scipy #ppf = percent point function, inverse of CDF(comulative distribution function) #also, CDF = pr(X<=x), i.e., probability to the left of the distribution z_critical = norm.ppf(1-a/2) #this returns a value for which the probab to the left is 0.975 p_value = 2*(1 - norm.cdf(np.abs(z))) p_value = float("{:.4f}".format(p_value)) print('z : ',z) print('\nz_critical :', z_critical) print('\nmean :', mean, "\n\n") #test the hypothesis if (np.abs(z) > z_critical): print("\nREJECT THE NULL HYPOTHESIS : \n p-value = ", p_value, "\n Alpha = ", a ) else: print("CANNOT REJECT THE NULL HYPOTHESIS. NOT ENOUGH EVIDENCE TO REJECT IT: \n p-value = ", p_value, "\n Alpha = ", a ) 

Answer: The .cdf() function calculates the probability for a given normal distribution value, while the .ppf() function calculates the normal distribution value for which a given probability is the required value. These are inverse of each other in this particular sense. To illustrate this calculation, check the below sample code. from scipy.stats import norm print(norm.ppf(0.95)) print(norm.cdf(1.6448536269514722))   This image with the code above should make it clear for you. Thanks!
Question: I am reading the "Deep Learning for Coders with fastai & PyTorch" book. I'm still a bit confused as to what the Embedding module does. It seems like a short and simple network, except I can't seem to wrap my head around what Embedding does differently than Linear without a bias. I know it does some faster computational version of a dot product where one of the matrices is a one-hot encoded matrix and the other is the embedding matrix. It does this to in effect select a piece of data? Please point out where I am wrong. Here is one of the simple networks shown in the book. class DotProduct(Module): def __init__(self, n_users, n_movies, n_factors): self.user_factors = Embedding(n_users, n_factors) self.movie_factors = Embedding(n_movies, n_factors) def forward(self, x): users = self.user_factors(x[:,0]) movies = self.movie_factors(x[:,1]) return (users * movies).sum(dim=1) 

Answer: Embedding  [...] what Embedding does differently than Linear without a bias.  Essentially everything. torch.nn.Embedding is a lookup table; it works the same as torch.Tensor but with a few twists (like possibility to use sparse embedding or default value at specified index). For example: import torch embedding = torch.nn.Embedding(3, 4) print(embedding.weight) print(embedding(torch.tensor([1])))  Would output: Parameter containing: tensor([[ 0.1420, -0.1886, 0.6524, 0.3079], [ 0.2620, 0.4661, 0.7936, -1.6946], [ 0.0931, 0.3512, 0.3210, -0.5828]], requires_grad=True) tensor([[ 0.2620, 0.4661, 0.7936, -1.6946]], grad_fn=<EmbeddingBackward>)  So we took the first row of the embedding. It does nothing more than that. Where is it used? Usually when we want to encode some meaning (like word2vec) for each row (e.g. words being close semantically are close in euclidean space) and possibly train them. Linear torch.nn.Linear (without bias) is also a torch.Tensor (weight) but it does operation on it (and the input) which is essentially: output = input.matmul(weight.t())  every time you call the layer (see source code and functional definition of this layer). Code snippet The layer in your code snippet does this:  creates two lookup tables in __init__ the layer is called with input of shape (batch_size, 2):  first column contains indices of user embeddings second column contains indices of movie embeddings   these embeddings are multiplied and summed returning (batch_size,) (so it's different from nn.Linear which would return (batch_size, out_features) and perform dot product instead of element-wise multiplication followed by summation like here)  This is probably used to train both representations (of users and movies) for some recommender-like system. Other stuff  I know it does some faster computational version of a dot product where one of the matrices is a one-hot encoded matrix and the other is the embedding matrix.  No, it doesn't. torch.nn.Embedding can be one hot encoded and might also be sparse, but depending on the algorithms (and whether those support sparsity) there might be performance boost or not.
Question: My filesystem (FS) (ZFS specifically) supports copy-on-write (COW), i.e. a copy (if done right) is a very cheap constant operation, and does not actually copy the underlying content. The content is copied only once I write/modify the new file. Actually, I just found out, ZFS-on-Linux actually has not implemented that for userspace yet (right?). But e.g. BTRFS or XFS has. (See here, here, here, here.) For the (GNU) cp utility, you would pass --reflink=always option (see here.) cp calls ioctl (dest_fd, FICLONE, src_fd) (see here, here). How would I get this behavior (if possible) in Python? I assume that "zero-copy" (e.g. here via os.sendfile) would not result in such behavior, right? Because looking at shutils _fastcopy_sendfile implementation (here), it is still a loop around os.sendfile using some custom byte count (supposed to be the block size, max(os.fstat(infd).st_size, 2 ** 23)). Or would it? The COW, is this on a file level, or block level? If possible, I want this to be generic and cross-platform as well, although my question here is somewhat Linux focused. A related question specifically about Mac seems to be this. The MacOSX cp has the -c option to clone a file.

Answer: While searching further, I actually found the answer, and a related issue report. Issue 37157 (shutil: add reflink=False to file copy functions to control clone/CoW copies (use copy_file_range)) is exactly about that, which would use FICLONE/FICLONERANGE on Linux. So I assume that shutil would support this in upcoming Python versions (maybe starting with Python 3.9?). There is os.copy_file_range (since Python 3.8), which wraps copy_file_range (Linux). However, according to issue 37159 (Use copy_file_range() in shutil.copyfile() (server-side copy)), Giampaolo Rodola:  Nope, [copy_file_range] doesn't [support CoW] (see man page). We can simply use FICLONE (cp does the same).  However, I'm not sure this is correct, as the copy_file_range man page says:  copy_file_range() gives filesystems an opportunity to implement "copy acceleration" techniques, such as the use of reflinks (i.e., two or more inodes that share pointers to the same copy- on-write disk blocks) or server-side-copy (in the case of NFS).  Issue 26826 (Expose new copy_file_range() syscall in os module) has this comment by Giampaolo Rodola:  I think data deduplication / CoW / reflink copy is better implemented via FICLONE. "cp --reflink" uses it, I presume because it's older than copy_file_range(). ...  Again, as noted already in the question, this does not work on ZFS yet, see this issue.
Question: I have looked for other questions, and this un-accepted-answered question is the only one I could find that somehow covers this issue and is not really helpful. Also, I need this to work with processes, and not threads. So from the ground up I wrote a sample program to show my issue, you should be able to paste it and it will run: import multiprocessing import time class Apple: def __init__(self, color): self.color = color def thinkAboutApple(apple): while True: print(apple.color) time.sleep(2) my_apple = Apple("red") new_process = multiprocessing.Process(target=thinkAboutApple, args=(my_apple,)) new_process.start() time.sleep(4) print("new: brown") my_apple.color = "brown" #so that the program doesn't exit after time.sleep(4) while True: pass  # actual output | # wanted output red | red red | red new: brown | new: brown red | brown red | brown  This tells me that either the apple is in a weird supposition where it is two colours at the same time, OR that the new_process' apple is in another position in ram and separated from the apple in the main process. So the question is: Is there a way to have the pointer of the apple in the process point to the same apple, or what is the pythonic way to keep all instances of the apple in all processes the same? What if I have the same apple in many processes and even more processes without the apple, how do I make sure they area always the same?

Answer: You can derive a specialized version of a Proxy class used by multiprocessing.BaseManager from the (undocumented) multiprocessing.managers.NamespaceProxy class that, unlike the base class, exposes all of its methods and attributes. This is similar to @shtse8's answer to the linked duplicate question, but I'm posting a runnable answer to it here to make clear how it can be done. from multiprocessing import Process from multiprocessing.managers import BaseManager, NamespaceProxy import time import types class MyManager(BaseManager): pass # Avoid namespace pollution. class Apple: def __init__(self, color): self.color = color def Proxy(target): """ Create a derived NamespaceProxy class for `target`. """ def __getattr__(self, key): result = self._callmethod('__getattribute__', (key,)) if isinstance(result, types.MethodType): def wrapper(*args, **kwargs): self._callmethod(key, args) return wrapper return result dic = {'types': types, '__getattr__': __getattr__} proxy_name = target.__name__ + "Proxy" ProxyType = type(proxy_name, (NamespaceProxy,), dic) # Create subclass. ProxyType._exposed_ = tuple(dir(target)) return ProxyType AppleProxy = Proxy(Apple) def thinkAboutApple(apple): while True: print(f"apple.color: {apple.color}") time.sleep(1) if __name__ == '__main__': MyManager.register('Apple', Apple, AppleProxy) manager = MyManager() manager.start() my_apple = manager.Apple("red") new_process = Process(target=thinkAboutApple, args=(my_apple,)) new_process.start() time.sleep(2) # Allow other process to run a short while. my_apple.color = "brown" # Change shared class instance. time.sleep(2) # Allow other process to run at little while longer. new_process.terminate()
Question: In NumPy, I would do a = np.zeros((4, 5, 6)) a = a[:, :, np.newaxis, :] assert a.shape == (4, 5, 1, 6)  How to do the same in PyTorch?

Answer: a = torch.zeros(4, 5, 6) a = a[:, :, None, :] assert a.shape == (4, 5, 1, 6)
Question: I have a bare bones project structure with mostly empty python files for the sake of testing a concept from an online tutorial: project |--package1 | |--__init__.py | |--module1.py | |--package2 | |--__init__.py | |--module2.py | |--__init__.py  module1.py: from .package2.module2 import function2  module2.py: def function2(): return 0  Running module1.py directly results in this error: Traceback (most recent call last): File "c:\"blahblahblah"\project\package1\module1.py", line 1, in <module> from .package2.module2 import function2 ImportError: attempted relative import with no known parent package  I've tried reducing the complexity of the issue by placing module2.py into the project folder itself and modifying the import as my tutorial suggests it would work (from .module2 import function2) but this yields the same error. side note: I am under the impression the init files are unnecessary for my version of python, but I've added them to keep all my bases covered. Python version 3.9.1 Any hints would be much appreciated.

Answer: The relative imports failed to work because module1.py could not look into it's parent folder for more packages when executed directly. The correct call required the -m parameter to signify that module1 was a module within a package. My terminal also needed to go up one directory: PS C:\...\"parent of project folder"> python -m project.package1.module1  Also, module1.py needed to be modified: from ..package2.module2 import function2
Question: I am learning how to use python by watching some online videos. When I run the code below using PyCharm, I get the following: macOS 11 or later required! Process finished with exit code 134 (interrupted by signal 6: SIGABRT) I have an M1 Mac mini with macOS Big Sur 11.1. This was happening when I had Python 3.8.2. Then, I installed Python 3.9.1. I keep getting the same error. How can I fix this? import turtle bob = turtle.Turtle() print(bob) 

Answer: If you used Homebrew to do your Python installation, there have been some reported issues with Python3 installs via brew (Source 1, Source 2). Updates are always in the works though, so you could try remedying your problem first with a brew update. If the issue persists, the current recommendation is to actually install Python directly from their site (Similar issue with fix reported). In conclusion, if using brew, try a brew update first, if that doesn't fix your issue, install Python directly! brew update && brew upgrade
Question: I just shifted from my old bud Sublime to VSCode. I really liked the way it works and the features it has. I'm a newbie python developer. I found two popular python extensions for VSCode: Python, and PyLance. My question is, What is the difference between Python and Pylance extension? I searched a lot but didn't find a good comparison.

Answer: As an editor, VSCode cannot recognize all languages and many functions cannot be implemented independently. Therefore, when we use Python code in VSCode, we need to install the 'Python' extension, which provides us with functions such as code completion, support for Jupyter notebooks, debugging Python code, etc. Therefore, the Python extension is one of the necessary dependencies for using Python in VSCode. The extension 'Pylance' needs to be used in conjunction with the Python extension. It cannot be used independently in VSCode. It mainly provides outstanding Python language services (other Python language services such as "Microsoft", "Jedi", don't need to install specific extensions, they can be used as-is after installation). At the same time, it also provides functions such as docstrings. Therefore, the Pylance extension is not a necessary condition, but a recommended extension. It is recommended that you install and use these two extensions. They are not opposite extensions, they are VSCode extensions that cooperate with each other (To be precise, the 'Pylance' extension relies on the 'Python' extension to use). And for more related information, you could refer to the VS Code docs on: Using Python in VSCode and Python and Pylance.
Question: I have just downloaded PyTorch with CUDA via Anaconda and when I type into the Anaconda terminal: import torch if torch.cuda.is_available(): print('it works')  then he outputs that; that means that it worked and it works with PyTorch. But when I go to my IDE (PyCharm and IntelliJ) and write the same code, it doesn't output anything. Could someone please explain to me how I can somehow get this to work in the IDE?

Answer: It was driving me mad as well... What finally helped me was the first link that says to use PyCharm "Terminal" to run the pip install command (from the PyTorch website). That fixed all my problems. (I had installed pytorch 3 times by that time and tried different interpreters...) https://www.datasciencelearner.com/how-to-install-pytorch-in-pycharm/ pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio===0.8.0 -f https://download.pytorch.org/whl/torch_stable.html I hope this helps save someone hours of headache. :)
Question: import twint import os, requests, re, time c = twint.Config() c.Username = <anyusername> #Replace with an actual uname in quotes c.Store_object = True c.Limit = 10 try: twint.run.Followers(c) except: print("Unexpected error:", sys.exc_info()[0]) f = twint.output.follows_list print(f)  Output CRITICAL:root:twint.feed:Follow:IndexError []  Have done pip install twint pip install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint  Googling, a lot of people have faced this error - but I cannot really find a solution Running the twint command line also gives the same error twint -u <uname> --followers CRITICAL:root:twint.feed:Follow:IndexError  This is not only followers. Anything I try, I get a similar error. Running Python 3.8.1 on Windows 10 Twint - latest version - 2.1.21

Answer:  Legacy mobile Twitter version will shut down on December 15th 2020. (M2 mobile web)  That's the headline of a reddit thread talking about the recent shutdown of the M2 Mobile Web ("Legacy") Twitter version; from this date on Twitter will only support these browsers. If you take a look at the GitHub repository of Twint: _usr.followers = int(ur['data']['user']['legacy']['followers_count'])  ... it uses the legacy attribute and therefore won't work from the 15th of December 2020. A contributor to the GitHub repo, namely @himanshudabas, also mentioned this in a reply to an identical issue:  That's because twitter killed the no js (mobile version) version of twitter on 15th December. Followers were scraped using that version.   You're not the only one facing this problem (obviously), and unfortunately with the Twint module there is no workaround (currently?). Speaking of the current situation you would need to either use the official Twitter API or alternative wrappers / unofficial APIs.
Question: I am trying to turn mp3 file to text, but my code returns the error outlined below. Any help is appreciated! This is a sample mp3 file. And below is what I have tried: import speech_recognition as sr print(sr.__version__) r = sr.Recognizer() file_audio = sr.AudioFile(r"C:\Users\Andrew\Podcast.mp3") with file_audio as source: audio_text = r.record(source) print(type(audio_text)) print(r.recognize_google(audio_text))  The full error I get. Appears to be: Error: file does not start with RIFF id  Thank you for your help!

Answer: You need to first convert the mp3 to wav, and then you can transcribe it, below is the modified version of your code. import speech_recognition as sr from pydub import AudioSegment # convert mp3 file to wav src=(r"C:\Users\Andrew\Podcast.mp3") sound = AudioSegment.from_mp3(src) sound.export("C:\Users\Andrew\podcast.wav", format="wav") file_audio = sr.AudioFile(r"C:\Users\Andrew\Podcast.wav") # use the audio file as the audio source r = sr.Recognizer() with file_audio as source: audio_text = r.record(source) print(type(audio_text)) print(r.recognize_google(audio_text))  In above modified code, first mp3 file being converted into wav and then transcribing processes.
Question: I'm figuring out how to return a list[] in asyncio I know asyncio.gather could help me but there are so many ways I'm now confused. How Do I return value from main() ? Thank async def wait_until(dt): # sleep until the specified datetime now = datetime.now() await asyncio.sleep((dt - now).total_seconds()) async def run_at(dt, coro): await wait_until(dt) return await coro async def main(): test=[] async for message in client.iter_messages(channel): test.append(message) return test loop = asyncio.get_event_loop() loop.create_task(run_at(datetime(2020, 12, 29, 19, 17),main())) loop.run_until_complete(asyncio.gather(*[main()])) # How to get test[] or How to pass it to another task? loop.run_forever() 

Answer: From the asyncio.gather documentation:  If all awaitables are completed successfully, the result is an aggregate list of returned values. The order of result values corresponds to the order of awaitables in aws.  From the asyncio.loop.run_until_complete documentation:  Return the Future’s result or raise its exception.  So gather is an async def that returns all the results passed, and run_until_complete runs the loop "converting" the awaitable into the result. Basically, the return values are passed through: results = loop.run_until_complete(asyncio.gather(*[main()])) tests = results[0]  Note that gather with just one item is redundant, as it's equivalent to just using that one item: tests = loop.run_until_complete(main())  If you want to communicate two independent tasks without using global variables, you probably want to use an asyncio.Queue, and give the same queue instance to both async def as input parameters. One will put "messages", and the other will get them. You can combine this with wait, gather, create_task, etc. to pretty much do everything you need.
Question: When making a request using the requests library to https://stackoverflow.com page = requests.get(url='https://stackoverflow.com') print(page.content)  I get the following: <!DOCTYPE html> <html class="html__responsive html__unpinned-leftnav"> <head> <title>Stack Overflow - Where Developers Learn, Share, &amp; Build Careers</title> <link rel="shortcut icon" href="https://cdn.sstatic.net/Sites/stackoverflow/Img/favicon.ico?v=ec617d715196"> <link rel="apple-touch-icon" href="https://cdn.sstatic.net/Sites/stackoverflow/Img/apple-touch-icon.png?v=c78bd457575a"> <link rel="image_src" href="https://cdn.sstatic.net/Sites/stackoverflow/Img/apple-touch-icon.png?v=c78bd457575a"> ..........  These source code here have the absolute paths, but when running the same URL using requests-html with js rendering with HTMLSession() as session: page = session.get('https://stackoverflow.com') page.html.render() print(page.content)  I get the following: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> <title>StackOverflow.org</title> <script type="text/javascript" src="lib/jquery.js"></script> <script type="text/javascript" src="lib/interface.js"></script> <script type="text/javascript" src="lib/window.js"></script> <link href="lib/dock.css" rel="stylesheet" type="text/css" /> <link href="lib/window.css" rel="stylesheet" type="text/css" /> <link rel="icon" type="image/gif" href="favicon.gif"/> ..........  The links here are relative paths, How can I get the source code with absolute paths like requests when using requests-html with js rendering?

Answer: This should probably a feature request for the request-html developers. However for now we can achieve this with this hackish solution: from requests_html import HTMLSession from lxml import etree with HTMLSession() as session: html = session.get('https://stackoverflow.com').html html.render() # iterate over all links for link in html.pq('a'): if "href" in link.attrib: # Make links absolute link.attrib["href"] = html._make_absolute(link.attrib["href"]) # Print html with only absolute links print(etree.tostring(html.lxml).decode())  We change the html-objects underlying lxml tree, by iterating over all links and changing their location to absolute using the html-object's private _make_absolute function.