eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe as follows: import pandas as pd df = pd.DataFrame({'sent.1':[0,1,0,1], 'sent.2':[0,1,1,0], 'sent.3':[0,0,0,1], 'sent.4':[1,1,0,1] })  I am trying to replace the non-zero values with the 5th character in the column names (which is the numeric part of the column names), so the output should be,  sent.1 sent.2 sent.3 sent.4 0 0 0 0 4 1 1 2 0 4 2 0 2 0 0 3 1 0 3 4  I have tried the following but it does not work, print(df.replace(1, pd.Series([i[5] for i in df.columns], [i[5] for i in df.columns])))  However when I replace it with column name, the above code works, so I am not sure which part is wrong. print(df.replace(1, pd.Series(df.columns, df.columns))) 

Answer: Since you're dealing with 1's and 0's, you can actually just use multiply the dataframe by a range: df = df * range(1, df.shape[1] + 1)  Output:  sent.1 sent.2 sent.3 sent.4 0 0 0 0 4 1 1 2 0 4 2 0 2 0 0 3 1 0 3 4  Or, if you want to take the numbers from the column names: df = df * df.columns.str.split('.').str[-1].astype(int)
Question: after installing Windows updates today, debugging is not working anymore. This is my active debug configuration: "launch": { "version": "0.2.0", "configurations": [ { "name": "DEBUG CURR", "type": "python", "request": "launch", "program": "${file}", "console": "internalConsole", "justMyCode": false, "stopOnEntry": false, }...  When I start the debugger, the menu pops up briefly for 1-2 seconds. But then it closes. There is no output in the console. It does not stop at set breakpoints. Does anybody have the same problem? Is there a solution? System settings  OS: Microsoft Windows 10 Enterprise (10.0.17763 Build 17763) VSCode version 1.64.0 Python version: 3.8.11 (in the active Anaconda Environment)  Installed VSCode extensions:  Python (Microsoft) version: v2022.0.1786462952 Pylance (Microsoft) version: v2022.2.0 

Answer: It's an issue with the latest Python Extension for VSCode. Downgrading the python extension to v2021.12.1559732655 fixes the problem.
Question: When running this code: embedding_matrix = torch.stack(embeddings)  I got this error:  RuntimeError: stack expects each tensor to be equal size, but got [7, 768] at entry 0 and [8, 768] at entry 1  I'm trying to get embedding using BERT via: split_sent = sent.split() tokens_embedding = [] j = 0 for full_token in split_sent: curr_token = '' x = 0 for i,_ in enumerate(tokenized_sent[1:]): token = tokenized_sent[i+j] piece_embedding = bert_embedding[i+j] if token == full_token and curr_token == '' : tokens_embedding.append(piece_embedding) j += 1 break sent_embedding = torch.stack(tokens_embedding) embeddings.append(sent_embedding) embedding_matrix = torch.stack(embeddings)  Does anyone know how I can fix this?

Answer: As per PyTorch Docs about torch.stack() function, it needs the input tensors in the same shape to stack. I don't know how will you be using the embedding_matrix but either you can add padding to your tensors (which will be a list of zeros at the end till a certain user-defined length and is recommended if you will train with this stacked tensor, refer this tutorial) to make them equidimensional or you can simply use something like torch.cat(data,dim=0).
Question: This is my first post and I am new to coding, so please let me know if you need more information. I have been running some AI to generate artwork and it has been working, but when I reloaded it the python script won't work and it is now saying "No module named 'transformers'". Can anyone help me out? It was when I upgraded to Google Colab Pro that I started to encounter issues although I am not sure why that would make a difference. ModuleNotFoundError 

Answer: Probably it is because you have not installed in your (new, since you've upgraded to colabs pro) session the library transformers. Try to run as first cell the following: !pip install transformers (the "!" at the beginning of the instruction is needed to go into "terminal mode" ). This will download the transformers package into the session's environment.
Question: Essentially, trying to write the following code results in the error below: Code from matplotlib import pyplot as plt plt.plot([1,2,3,2,1]) plt.show()  Error libGL error: MESA-LOADER: failed to open iris: /home/xxx/.conda/envs/stat/lib/python3.8/site-packages/pandas/_libs/window/../../../../../libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib64/dri/iris_dri.so) (search paths /usr/lib64/dri, suffix _dri) libGL error: failed to load driver: iris libGL error: MESA-LOADER: failed to open swrast: /home/xxx/.conda/envs/stat/lib/python3.8/site-packages/pandas/_libs/window/../../../../../libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib64/dri/swrast_dri.so) (search paths /usr/lib64/dri, suffix _dri) libGL error: failed to load driver: swrast  I found similar errors on StackOverflow but none were what is needed here.

Answer: Short answer: export LD_PRELOAD=/usr/lib64/libstdc++.so.6 Long answer: The underlying problem is that we have a piece of software that was built with an older C++ compiler. Part of the compiler is its implementation of libstdc++ which becomes part of the runtime requirements for anything built by the compiler. The software in question has, evidently, brought its own, older implementation of libstdc++ along for the ride, and given its libstdc++ precedence over the system's libstdc++. Typically, this is done via the $LD_LIBRARY_PATH environment variable. Unfortunately, /usr/lib64/dri/swrast_dri.so is a piece of system software built by the native compiler for that system, and it's more recent than the compiler that built the other software in question. The result of this is that the older compiler's libstdc++ gets loaded first, with its older, more limited symbol set. When it then wants to load swrast, this fails because swrast insists on having the level of compiler/runtime with which it was built. The solution to this whole mess is the force the system's (newer) libstdc++ into use and prevent the older libstdc++ from being brought into play. This is achieved via the code snippet export LD_PRELOAD=/usr/lib64/libstdc++.so.6 where we set the preload environment variable.
Question: I want to know if there is a way to have a default value in a dictionary (without using the get function), so that: dict colors = { "Black":(0, 0, 0), "White":(255, 255, 255), default:(100, 100, 100) }; paint(colors["Blue"]); # Paints the default value (Grey) onto the screen  Of course, the code above wouldn't work in Python, and I have serious doubt it is even possible. I Know that by using get I could do this easily, however I'm still curious if there is any other way (just for curiosity).

Answer: You can use a defaultdict. from collections import defaultdict colors = defaultdict(lambda: (100, 100, 100)) colors["Black"] = (0, 0, 0), colors["White"] = (255, 255, 255) # Prints (0, 0, 0), because "Black" is mapped to (0, 0, 0) in the dictionary. print(colors["Black"]) # Prints (100, 100, 100), because "Blue" is not a key in the dictionary. print(colors["Blue"])
Question: My CLI applications typically have subcommands. I want to have the --version flag at the root of my CLI applications, but with Typer I've only seen ways to put it to a command. I want to add it to the typer.Typer object (the root) itself. How can I do that? What I've tried import typer from typing import Optional __version__ = "0.1.0" def version_callback(value: bool): if value: typer.echo(f"Awesome CLI Version: {__version__}") raise typer.Exit() app = typer.Typer( add_completion=False, ) @app.command() def main( version: Optional[bool] = typer.Option( None, "--version", callback=version_callback ), ) -> None: pass @app.command() def foo() -> None: pass if __name__ == "__main__": app()  This gives $ python cli.py --help Usage: cli.py [OPTIONS] COMMAND [ARGS]... Options: --help Show this message and exit. Commands: foo main $ python cli.py main --help Usage: cli.py main [OPTIONS] Options: --version --help Show this message and exit.  What I wanted: $ python cli.py --help Usage: cli.py [OPTIONS] COMMAND [ARGS]... Options: --version --help Show this message and exit. Commands: foo 

Answer: This is addressed in the documentation:  But as those CLI parameters are handled by each of those commands, they don't allow us to create CLI parameters for the main CLI application itself. But we can use @app.callback() for that. It's very similar to @app.command(), but it declares the CLI parameters for the main CLI application (before the commands):  To do what you want, you could write something like this: import typer from typing import Optional __version__ = "0.1.0" def version_callback(value: bool): if value: typer.echo(f"Awesome CLI Version: {__version__}") raise typer.Exit() app = typer.Typer( add_completion=False, ) @app.callback() def common( ctx: typer.Context, version: bool = typer.Option(None, "--version", callback=version_callback), ): pass @app.command() def main() -> None: pass @app.command() def foo() -> None: pass if __name__ == "__main__": app()  Which gives us: $ python typertest.py --help Usage: typertest.py [OPTIONS] COMMAND [ARGS]... Options: --version --help Show this message and exit. Commands: foo main
Question: i need to do a script for imap backup but when i'm trying to connect to the imap server with my script i'm getting that error:  File "c:\Users\Lenovo\Desktop\python\progettoscuola.py", line 5, in <module> imapSrc = imaplib.IMAP4_SSL('mail.safemail.it') File "C:\Program Files\Python310\lib\imaplib.py", line 1323, in __init__ IMAP4.__init__(self, host, port, timeout) File "C:\Program Files\Python310\lib\imaplib.py", line 202, in __init__ self.open(host, port, timeout) File "C:\Program Files\Python310\lib\imaplib.py", line 1336, in open IMAP4.open(self, host, port, timeout) File "C:\Program Files\Python310\lib\imaplib.py", line 312, in open self.sock = self._create_socket(timeout) File "C:\Program Files\Python310\lib\imaplib.py", line 1327, in _create_socket return self.ssl_context.wrap_socket(sock, File "C:\Program Files\Python310\lib\ssl.py", line 512, in wrap_socket return self.sslsocket_class._create( File "C:\Program Files\Python310\lib\ssl.py", line 1070, in _create self.do_handshake() File "C:\Program Files\Python310\lib\ssl.py", line 1341, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:997)``` 

Answer: Python 3.10 increased the default security settings of the TLS stack by among other things prohibiting any ciphers which still use the RSA key exchange. RSA key exchange is long considered inferior since it does not provide forward secrecy and is therefore also no longer available in TLS 1.3. So in general the change in Python 3.10 can be considered an improvement. But, some servers still require this obsolete key exchange and mail.safemail.it seems to be among these. Connecting to such servers with the newly hardened TLS settings will now fail, even if it succeeded with older versions of Python. To make connections possible again it is necessary to use weaker security settings. For this specific server it can be done by falling back to the DEFAULT ciphers used by OpenSSL. The following code will create a new SSL context and use it for connecting to the host. The important part here is to use weaker settings using ctx.set_ciphers('DEFAULT') . import imaplib import ssl ctx = ssl.create_default_context() ctx.set_ciphers('DEFAULT') imapSrc = imaplib.IMAP4_SSL('mail.safemail.it', ssl_context = ctx)
Question: I am trying to understand how dot product works for dimensions more than 2. The documentation says:  If a is an N-D array and b is an M-D array (where M>=2), it is a sum product over the last # axis of a and the second-to-last axis of b: dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])   I don't understand this rule -- where does it come from? Why is it the last axis of a and the second-to-last axis of b?

Answer: It may be easier to visualize this using the notation of np.einsum. Start with regular 2D matrix multiplication, which is pretty unambiguous, and follows "normal math" rules: a = np.ones((2, 3)) b = np.ones((3, 4)) np.einsum('ij,jk->ik', a, b) # Same as a.dot(b)  Now prepend a few dimensions: a = np.ones((2, 3, 4, 5)) b = np.ones((8, 7, 5, 6)) np.einsum('ijkl,nolp->ijknop', a, b) # Same as a.dot(b)  In general, a multidimensional array can be thought of as a stack of matrices. There are a number of reasons that numpy generally places the "elements" of an array, whether features, samples, vectors, or matrices along the last dimensions. So, regardless of dot, it is fairly standard to think of a as a 2x3 array of 4x5 matrices. Similarly, b is a 8x7 array of 5x6 matrices. There are two options going forward. The last two dimensions of the result are unambiguous no matter what choice you make: the output will contain 4x6 matrices. You can compute all the possible combinations of 2x3x8x7 such matrices, or you can expect the leading dimensions to broadcast together. np.dot takes the former approach, while @/np.matmul takes the latter. Anecdotally, I've found that most people will find the latter more intitive. The exact reason that np.dot computes the matrix product of all possible combinations of a and b is known only to the developers making that choice. It likely revolves around not wanting to raise an error, and the rules of broadcasting not being as universally cemented at the time dot was first implemented. Once you decide to take the products of all the combinations, you can decide to do it like this: np.einsum('ijkl,nolp->ijnokp', a, b) # Not `np.dot`  On the one hand, this would reflect the idea of a 2x3x8x7 array of 4x6 matrices better. On the other hand, it creates an awkward permutation of dimensions with k out of place in the sequence. The developers went with the arguably less awkward approach of grouping the dimensions from each array separately. You can always a transpose an array to the shape you want after all.
Question: (The following code can be run in Jupyter.) I have a class B, which uses class A, needs to be tested. class A: async def f(self): pass class B: async def f(self): a = A() x = await a.f() # need to be patched/mocked  And I have the following test code. It seems it mocked the class method of A instead of the instance method. from asyncio import Future from unittest.mock import MagicMock, Mock, patch async def test(): sut = B() with patch('__main__.A') as a: # it's __main__ in Jupyter future = Future() future.set_result('result') a.f = MagicMock(return_value=future) await sut.f() await test()  However, the code got the error of: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\X~1\AppData\Local\Temp\1/ipykernel_36576/3227724090.py in <module> 20 await sut.f() 21 ---> 22 await test() C:\Users\X~1\AppData\Local\Temp\1/ipykernel_36576/3227724090.py in test() 18 future.set_result('result') 19 a.f = MagicMock(return_value=future) ---> 20 await sut.f() 21 22 await test() C:\Users\X~1\AppData\Local\Temp\1/ipykernel_36576/3227724090.py in f(self) 6 async def f(self): 7 a = A() ----> 8 x = await a.f() # need to be patched/mocked 9 10 from asyncio import Future TypeError: object MagicMock can't be used in 'await' expression 

Answer: In Python 3.8+, patching an async method gives you an AsyncMock, so providing a result is a little more straightforward. In the docs of the patch method itself:  If new is omitted, then the target is replaced with an AsyncMock if the patched object is an async function or a MagicMock otherwise.  AsyncMock lets you supply a return value in a much more straightforward manner: import asyncio from unittest.mock import patch class A: async def f(self): return "foo" class B: async def f(self): return await A().f() async def main(): print(await B().f()) with patch("__main__.A.f", return_value="bar") as p: print(await B().f()) if __name__ == "__main__": try: asyncio.run(main()) except KeyboardInterrupt: sys.exit(1)  ....prints: $ python example.py foo bar  The side_effect kwarg covers most kinds of values you'd be looking to return (e.g. if you need your mock function await something).   if side_effect is a function, the async function will return the result of that function, if side_effect is an exception, the async function will raise the exception, if side_effect is an iterable, the async function will return the next value of the iterable, however, if the sequence of result is exhausted, StopAsyncIteration is raised immediately, if side_effect is not defined, the async function will return the value defined by return_value, hence, by default, the async function returns a new AsyncMock object.
Question: I ran the following code in RStudio: exo <- read.csv('exoplanets.csv',TRUE,",") df <- data.frame(exo) ranks <- 570 files <- 3198 datas <- vector() for ( w in 2:files ) { listas <-vector() for ( i in 1:ranks) { name <- as.character(df[i,w]) listas <- append (listas, name) } datas <- append (datas, listas) }  It reads a huge NASA CSV file, converts it to a dataframe, converts each element to string, and adds them to a vector. RStudio took 4 min and 15 seconds. So I decided to implement the same code in Julia. I ran the following in VS Code: using CSV, DataFrames df = CSV.read("exoplanets.csv", DataFrame) fil, col = 570, 3198 arr = [] for i in 2:fil for j in 1:col push!(arr, string(df[i, j])) end end  The result was good. The Julia code took only 1 minute and 25 seconds! Then for pure curiosity I implemented the same code this time in Python to compare. I ran the following in VS Code: import numpy as np import pandas as pd exo = pd.read_csv("exoplanets.csv") arr = np.array(exo) fil, col = 570, 3198 lis = [] for i in range(1, fil): for j in range(col): lis.append(arr[i][j].astype('str'))  The result shocked me! Only 35 seconds!!! And in Spyder from Anaconda only 26 seconds!!! Almost 2 million floats!!! Is Julia slower than Python in data analysis? Can I improve the Julia code?

Answer: NOTE: I wrote the below assuming you want the other column order (as in the Python and R examples). It is more efficient in Julia this way; to make it work equivalently to your original behaviour, permute the logic or your data at the right places (left as an exercise). Bogumił's anwer does the right thing already.  Put stuff into functions, preallocate where possible, iterate in stride order, use views, and use builtin functions and broadcasting: function tostringvector(d) r, c = size(d) result = Vector{String}(undef, r*c) v = reshape(result, r, c) for (rcol, dcol) in zip(eachcol(v), eachcol(d)) @inbounds rcol .= string.(dcol) end return result end  Which certainly can be optimized harder. Or shorter, making use of what DataFrames already provides: tostringvector(d) = vec(Matrix(string.(d)))
Question: I am trying to create a bar chart with year data on x-axis. It works but the year marker on x-xais are all in vertical direction and I want to make them more readable - either horizontal or 45 degree. I tried using the year:T in datatime format but it gave me the year and month markers (I just wanted to have the year markers on xais). How do I make the year markers on x-axis either horizontal or 45 degree? import altair as alt from vega_datasets import data source = data.wheat() source.info() abars = alt.Chart(source).mark_bar().encode( x='year:O', y="wheat:Q" )  

Answer: Set labelAngle: alt.Chart(source).mark_bar().encode( x=alt.X('year:O', axis=alt.Axis(labelAngle=-45)), y="wheat:Q" )
Question: I'm dealing with JSON data which I load into Python dictionaries. A lot of these have optional fields, which then may contain dictionaries, that kind of stuff. dictionary1 = {"required": {"value1": "one", "value2": "two"}, "optional": {"value1": "one"}} dictionary2 = {"required": {"value1": "one", "value2": "two"}}  If I do this, dictionary1.get("required").get("value1")  this works, obviously, because the field "required" is always present. However, when I use the same line on dictionary2 (to get the optional field), this will produce an AttributeError dictionary2.get("optional").get("value1") AttributeError: 'NoneType' object has no attribute 'get'  which makes sense, because the first .get() will return None, and the second .get() cannot call .get() on the None object. I can solve this by giving default values in case the optional field is missing, but this will be annoying the more complex the data gets, so I'm calling this a "naive fix": dictionary2.get("optional", {}).get("value1", " ")  So the first .get() will return an empty dictionary {}, on which the second .get() can be called, and since it obviously contains nothing, it will return the empty string, as defined per the second default. This will no longer produce errors, but I was wondering if there is a better solution for this - especially for more complex cases (value1 containing an array or another dictionary, etc....) I could also fix this with try - except AttributeError, but this is not my preferred way either. try: value1 = dictionary2.get("optional").get("value1") except AttributeError: value1 = " "  I also don't like checking if optional field exists, this produces garbage code lines like optional = dictionary2.get("optional") if optional: value1 = optional.get("value1") else: value1 = " "  which seems very non-Pythonic... I was thinking maybe my approach of just chaining .get()s is wrong in the first place?

Answer: First of all, you refer to " " as the empty string. This is incorrect; "" is the empty string. Second, if you're checking for membership, I don't see a reason to use the get method in the first place. I'd opt for something like the following. if "optional" in dictionary2: value1 = dictionary2["optional"].get("value1") else: value1 = ""   Another alternative to consider (since you're using the get method a lot) is to switch to the defaultdict class. For example, from collections import defaultdict dictionary2 = {"required": {"value1": "one", "value2": "two"}} ddic2 = defaultdict(dict,dictionary2) value1 = ddic2["optional"].get("value1")
Question: Can anyone explain what the -svv flag means/does when running pytest from command line? Such as: pytest -svv  This is really driving me crazy, as I can't find it in pytest's official documentation, its code base or through a web search.

Answer: This is the same as passing pytest -s -vv  where -s disables capturing of stdout/stderr (source) -vv enables verbose output (source) Specifically note that the increased verbosity specifier currently has no effect in base pytest above normal increased verbosity  Using higher verbosity levels (-vvv, -vvvv, …) is supported, but has no effect in pytest itself at the moment, however some plugins might make use of higher verbosity
Question: I'm having some issues combining single-dispatch overloads with a protocol for structural typing based on having a specific attribute. I have constructed the following snippet to give an idea of what I'm attempting. from typing_extensions import Protocol from functools import singledispatch class HasFoo(Protocol): foo: str class FooBar: def __init__(self): self.foo = 'bar' @singledispatch def f(_) -> None: raise NotImplementedError @f.register def _(item: HasFoo) -> None: print(item.foo) x = FooBar() f(x)  The above raises the NotImplementedError instead of the print statement as desired. I've tried various modifications to the above with no success. Any help would be very appreciated.

Answer: That apparently doesn't work. Single dispatch uses the MRO (Method resolution order) to find out if a given instance matches a registered type. However, the protocol is not part of the MRO, hence python will not find it. To make it part of the MRO you could inherit from it, but that is not what Protocols are meant to be used for (though I think it also doesn't cost anything).