eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an array: array([nan, 'Stressful day', 'Drank coffee:Drank tea', 'Drank tea', 'Ate late:Drank coffee', 'Drank coffee:Drank tea:Worked out', 'Drank tea:Worked out', 'Drank coffee:Drank tea:Stressful day', 'Drank coffee', 'Drank coffee:Drank tea:Stressful day:Worked out', 'Drank coffee:Worked out', 'Ate late:Drank coffee:Drank tea', 'Ate late:Drank coffee:Drank tea:Worked out', 'Drank tea:Stressful day', 'Drank tea:Stressful day:Worked out', 'Drank coffee:Stressful day:Worked out', 'Drank coffee:Stressful day', 'Ate late:Drank coffee:Drank tea:Stressful day', 'Worked out', 'Ate late:Drank coffee:Worked out'], dtype=object)  these are unique values from the column of a dataframe, as you can see they are combination of other values like 'Drank coffee:Drank tea' is a combination of 'Drank coffee' and 'Drank tea'. I want those unique elements for this list. What's the quickest way to create that list? Is there any inbuilt function in python libraries for this sort of thing? Expected output: array([nan, 'Stressful day', 'Drank coffee', 'Drank tea', 'Ate late', 'Worked out'], dtype=object) 

Answer: Assuming a the input array, you could use str.extractall: out = pd.Series(a).str.extractall('([^:]+)')[0].unique()  From the original Series s: out = s.unique().drop_duplicates().str.extractall('([^:]+)')[0].unique()  Output: array(['Stressful day', 'Drank coffee', 'Drank tea', 'Ate late', 'Worked out'], dtype=object)  Other options (maybe less efficient): out = set(x for s in a if isinstance(s, str) for x in s.split(':')) out = pd.Series(a).str.split(':').explode().unique()  keeping NaNs: s = pd.Series(a) out = np.concatenate([s[s.isna()].unique(), s.str.extractall('([^:]+)')[0].unique()])  Output: array([nan, 'Stressful day', 'Drank coffee', 'Drank tea', 'Ate late', 'Worked out'], dtype=object)  Or: out = set(x for s in a for x in (s.split(':') if isinstance(s, str) else [s]))  Output: {'Drank coffee', 'Drank tea', nan, 'Stressful day', 'Worked out', 'Ate late'}
Question: I have a script that up until recently worked fine, but is now producing an error. import requests import pandas as pd # Set the url to given endpoint url = "https://SomeURL/SomeEndpoint" print('URL set') # Connect to endpoint with credentials and put results in dictionary URLresponse = requests.get(url,auth=("SomeUser", "SomePassword"), verify=True) print('connection to endpoint') # Load the response as proper JSON into a var rawdata = (URLresponse.content) print(type(rawdata)) print('populating variable') # print(rawdata) # Load the var into a dataframe df = pd.read_json(rawdata) print('load variable into df') print(df)  This used to work fine but now it is producing an error as following:  File "C:\Program Files\Python310\lib\site-packages\pandas\io\common.py", line 901, in get_handle raise TypeError( TypeError: Expected file path name or file-like object, got <class 'bytes'> type  How can I go ahead to troubleshoot this?

Answer: You can change df = pd.read_json(rawdata) to df = pd.read_json(io.StringIO(rawdata.decode('utf-8'))) You will need to include import io earlier in your file as well.
Question: I am working with a list of dictionaries in Python that represents a set of data records. The data structure looks like this: data = [ {'Name': 'Tom', 'Age': 25, 'Score': 85}, {'Name': 'Alex', 'Age': 30, 'Score': 80}, {'Name': 'Tom', 'Age': 20, 'Score': 90}, {'Name': 'Alex', 'Age': 25, 'Score': 95}, {'Name': 'Tom', 'Age': 25, 'Score': 80}, {'Name': 'Alex', 'Age': 30, 'Score': 85} ]  Each dictionary in the list represents a single record with 'Name', 'Age', and 'Score' as fields. My goal is to sort this list of dictionaries according to multiple fields. I want to sort it first by the 'Name' field in alphabetical order, then by the 'Age' field in ascending numerical order, and finally by the 'Score' field in descending numerical order. I have been trying to use the sorted() function in Python, but I can't figure out how to sort by multiple fields at once, and especially not how to sort by different fields in different directions. I would like to know the most efficient way to achieve this multi-level sort in Python. Any suggestions would be greatly appreciated.

Answer: You can use key= parameter in sorted() or .sort(). The key parameter will return 3-item tuple, where the score is negated (to have it in descending order): data.sort(key=lambda d: (d["Name"], d["Age"], -d["Score"])) print(data)  Prints: [ {"Name": "Alex", "Age": 25, "Score": 95}, {"Name": "Alex", "Age": 30, "Score": 85}, {"Name": "Alex", "Age": 30, "Score": 80}, {"Name": "Tom", "Age": 20, "Score": 90}, {"Name": "Tom", "Age": 25, "Score": 85}, {"Name": "Tom", "Age": 25, "Score": 80}, ]
Question: Let's say I have a 3d numpy array.shape of (27,27,27). I want to compress this to (9,9,9) by averaging every 3 elements across every axis simultaneously (e.g. make 3x3x3 pixels into 1x1x1). The objective is to effectively compress by a single integer across all three axes simultaneously (with the assumption that any array will have a multiple of that integer for the shape of each axes). My initial attempt was to use np.apply_over_axes, though I'm worried it is not getting the cubic mean of all 3 axes but instead averaging each sequentially. def mean_over(arr, axis): np.average(arr.reshape(-1, 3), axis=axis) the_array_small = np.apply_over_axes(mean_over, the_array, \[0,1,2\])  However this returns an error: Traceback (most recent call last): File "\<stdin\>", line 1, in \<module\> File "\<\__array_function_\_ internals\>", line 180, in apply_over_axes File "/opt/homebrew/Caskroom/mambaforge/base/envs/seaborn/lib/python3.10/site-packages/numpy/lib/shape_base.py", line 496, in apply_over_axes if res.ndim == val.ndim: AttributeError: 'NoneType' object has no attribute 'ndim'  I'm not convinced my apply_over_axes solution gets the aggregation I'm aiming for though. Ideally the mean of each (3,3,3) component is returned.

Answer: Another solution but take advantage of as_strided a = np.arange(27**3).reshape(27, 27, 27) tile_size = (3, 3, 3) tile_shape = tuple(np.array(a.shape) // np.array(tile_size)) tile_strides = tuple(np.array(a.strides) * np.array(tile_size)) + tuple(a.strides) tile_view = np.lib.stride_tricks.as_strided( a, shape=tile_shape + tile_size, strides=tile_strides, writeable=False, ) result = np.mean(tile_view, axis=(-3, -2, -1))
Question: I am trying to put together a simple "Q&A with sources" using Langchain and a specific URL as the source data. The URL consists of a single page with quite a lot of information on it. The problem is that RetrievalQAWithSourcesChain is only giving me the entire URL back as the source of the results, which is not very useful in this case. Is there a way to get more detailed source info? Perhaps the heading of the specific section on the page? A clickable URL to the correct section of the page would be even more helpful! I am slightly unsure whether the generating of the result source is a function of the language model, URL loader or simply RetrievalQAWithSourcesChain alone. I have tried using UnstructuredURLLoader and SeleniumURLLoader with the hope that perhaps more detailed reading and input of the data would help - sadly not. Relevant code excerpt: llm = ChatOpenAI(temperature=0, model_name='gpt-3.5-turbo') chain = RetrievalQAWithSourcesChain.from_llm(llm=llm, retriever=VectorStore.as_retriever()) result = chain({"question": question}) print(result['answer']) print("\n Sources : ",result['sources'] ) 

Answer: ChatGPT is very flexible, and the more explicit you are better results you can get. This link show the docs for the function you are using. there is a parameter for langchain.prompts.BasePromptTemplate that allows you to give ChatGPT more explicit instructions. It looks like the base prompt template is this  Use the following knowledge triplets to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n{context}\n\nQuestion: {question}\nHelpful Answer:  You can add in another sentence giving ChatGPT more clear instructions  Please format the answer with JSON of the form { "answer": "{your_answer}", "relevant_quotes": ["list of quotes"] }. Substitutde your_answer as the answer to the question, but also include relevant quotes from the source material in the list.  You may need to tweak it a little bit to get ChatGPT responding well. Then you should be able to parse it. ChatGPT has 3 message types in the API  User - a message from an end user to the model model - a message from the model to the end user system - a message from the prompt engineer to model to add instructions. Lang chain doesn't use this since it's a one-shot prompt  I strongly recommend these courses on ChatGPT since they are from Andrew Ng and very high quality.
Question: I am confused as to what the "@" operator actually does in poetry add pandas@^1.3.0. Both following commands install pandas version 1.5.3 and set the dependency in my pyproject.toml to pandas = "^1.3.0": poetry add pandas@^1.3.0 poetry add pandas^1.3.0  I have no other dependencies listed (aside from Python 3.8). I thought that using the "@" symbol signifies a strict requirement for a specific version and its compatible releases. With "pandas@^1.3.0," shouldn't Poetry install exactly the version 1.3.0 of the "pandas" package? The official documentation says:  When adding dependencies via poetry add, you can use the @ operator. This is understood similarly to the == syntax, but also allows prefixing any specifiers that are valid in pyproject.toml. For example: 

Answer: The "@" operator in the add command is a delimiter between the package name and the version. If the "@" operator is followed by its required version, e.g. poetry add pendulum@2.0.5  is the same as: poetry add pendulum==2.0.5  If you use caret, e.g.: poetry add requests@^2.13.0  Then you specify a version range. The "@" symbol signifies a strict requirement only if you don't use any other specifier afterwards. The poetry documentation for "@" operator is a bit confusing.
Question: I have an array A. I am identifying ones in each row except the row number itself and creating a list. For example, in A[0], the ones should be identified for locations 2,3,5 and not 0. I present the current and expected output. import numpy as np A=np.array([[1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0], [1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0], [1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0], [0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1], [0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0], [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1], [0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0], [0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1]]) output = [] for i, row in enumerate(A): ones_indices = np.where(row == 1)[0] other_rows = np.arange(A.shape[0]) other_rows = np.delete(other_rows, i) output.append([[i], other_rows.tolist()]) print(output)  The current output is [[[0], [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]], [[1], [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]], [[2], [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11]], [[3], [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11]], [[4], [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11]], [[5], [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11]], [[6], [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11]], [[7], [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11]], [[8], [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11]], [[9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11]], [[10], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11]], [[11], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]]  The expected output is [[[0], [2,3,5]], [[1], [3,4,6]], [[2], [0,3,5]], [[3], [0, 1, 2, 4, 5, 6]], [[4], [1,3,6]], [[5], [0,2,3,7,8,10]], [[6], [1,3,4,8,9,11]], [[7], [5,8,10]], [[8], [5,6,7,9,10,11]], [[9], [6,8,11]], [[10], [5,7,8], [[11], [6,8,9]]] 

Answer: The numpy approach would be to fill_diagonal, then to use where: np.fill_diagonal(A, 0) row, idx = np.where(A==1) # np.where(A) if only 0/1  Output: (array([ 0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11]), array([ 2, 3, 5, 3, 4, 6, 0, 3, 5, 0, 1, 2, 4, 5, 6, 1, 3, 6, 0, 2, 3, 7, 8, 10, 1, 3, 4, 8, 9, 11, 5, 8, 10, 5, 6, 7, 9, 10, 11, 6, 8, 11, 5, 7, 8, 6, 8, 9]))  If you really want nested lists: np.fill_diagonal(A, 0) out = [[[i], np.where(a==1)[0].tolist()] for i, a in enumerate(A)]  Or: row, idx = np.where(A==1) x = np.array_split(idx, np.where(row[1:]!=row[:-1])[0]+1) out = [[[i], j.tolist()] for i,j in enumerate(x)]  Output: [[[0], [2, 3, 5]], [[1], [3, 4, 6]], [[2], [0, 3, 5]], [[3], [0, 1, 2, 4, 5, 6]], [[4], [1, 3, 6]], [[5], [0, 2, 3, 7, 8, 10]], [[6], [1, 3, 4, 8, 9, 11]], [[7], [5, 8, 10]], [[8], [5, 6, 7, 9, 10, 11]], [[9], [6, 8, 11]], [[10], [5, 7, 8]], [[11], [6, 8, 9]]]
Question: Let's take two sets: a = [5,5,5,5,5,4,4,4,4,3,3,3,2,2,1] b = [5,4,3,2,1]  We perform the KS-Test using Python: from scipy import stats stats.ks_2samp(b,a) KstestResult(statistic=0.2, pvalue=0.9979360165118678, statistic_location=2, statistic_sign=1)  Why is the result a p-value of 0.9979? This means that the distribution of the values in the two sets is almost identical. But it's not! What do I missunderstand? Kind regards.

Answer: The observed value of the KS test statistic, namely 0.2, is actually relatively small, considering the distribution of the test statistic for a reasonable null hypothesis; I think this is where the surprise is coming from. As mentioned, the usual KS test assumes there are no ties, so we'll have to compute the p-value ourselves. We can make progress by assuming the null hypothesis is that both samples come from a uniform distribution and estimating the p-value by random sampling. (This null hypothesis is more restrictive than the conventional one which just assume the same distribution, not necessarily uniform.) Here are a few lines of R code to estimate the p-value. In the interest of brevity, it's specific to the problem as stated: a sample of size 5 and a sample of size 15, each one from a uniform distribution on the set { 1, 2, 3, 4, 5 }, and the observed KS test statistic is 0.2. my.ecdf <- function (x) cumsum (sapply (1:5, function (k) sum (x == k))/length(x)) R <- function (n) sample.int (5, size = n, replace = T) generate.ks.test.statistic <- function (n) sapply (1:n, function (k) max (abs (my.ecdf (R (15)) - my.ecdf (R (5))))) ks <- generate.ks.test.statistic (10000) sum (ks >= 0.2)/10000  For this last input, I get 0.8243. That's not as extreme as the value you reported (more than 0.99), but still enough to show that 0.2 is actually relatively small. You can look at hist(ks) to see what the distribution looks like.
Question: I've just started with Selenium and I'm already stuck at the first step: setting up the driver. I keep getting this error:  'str' object has no attribute '_ignore_local_proxy'.  Here's the code : from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager import requests driver = webdriver.Chrome(ChromeDriverManager().install())  And the whole traceback : AttributeError Traceback (most recent call last) Cell In[21], line 1 ----> 1 driver = webdriver.Chrome(ChromeDriverManager().install()) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\chrome\webdriver.py:49, in WebDriver.__init__(self, options, service, keep_alive) 45 self.keep_alive = keep_alive 47 self.service.path = DriverFinder.get_path(self.service, self.options) ---> 49 super().__init__( 50 DesiredCapabilities.CHROME["browserName"], 51 "goog", 52 self.options, 53 self.service, 54 self.keep_alive, 55 ) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\chromium\webdriver.py:60, in ChromiumDriver.__init__(self, browser_name, vendor_prefix, options, service, keep_alive) 51 self.service.start() 53 try: 54 super().__init__( 55 command_executor=ChromiumRemoteConnection( 56 remote_server_addr=self.service.service_url, 57 browser_name=browser_name, 58 vendor_prefix=vendor_prefix, 59 keep_alive=keep_alive, ... 63 ) 64 except Exception: 65 self.quit() AttributeError: 'str' object has no attribute '_ignore_local_proxy'  I'm using VS Code with Python 3.11, if that can somehow help.

Answer: This is due to changes in selenium 4.10.0: https://github.com/SeleniumHQ/selenium/commit/9f5801c82fb3be3d5850707c46c3f8176e3ccd8e  Note that the first argument is no longer executable_path, but options. (ChromeDriverManager().install() returns the path to the install location.) Since selenium manager is now included with selenium 4.10.0, you should no longer use ChromeDriverManager at all. from selenium import webdriver driver = webdriver.Chrome() # ... driver.quit()  However, if you still want to pass in the executable_path to an existing driver, you must use the service arg now: from selenium import webdriver from selenium.webdriver.chrome.service import Service service = Service(executable_path="PATH_TO_DRIVER") options = webdriver.ChromeOptions() driver = webdriver.Chrome(service=service, options=options) # ... driver.quit()
Question: I've come across a bug that I can't seem to understand. I have a tkinter Text widget that has a bind that triggers on text modification. For some reason this event gets triggered when I use the key combination even though it shouldn't, as it doesn't modify the contents of the Text widget. Here comes the weird part: this only occurs with <Control-o>. I have made a simple program to demonstrate the problem. Other than special preassigned key combinations such as <Control-i> that actually modify the content, no other combination behaves like this. Why does this occur for <Control-o> specifically? And how do I prevent it? import tkinter as tk root = tk.Tk() txt = tk.Text(root) txt.pack() root.bind("<Control-u>", lambda e: print("doesn't trigger")) root.bind("<Control-o>", lambda e: print("somehow triggers")) txt.bind("<<Modified>>", lambda e: print("text got modified!")) # (keep in mind that this will only get triggered once) 

Answer: The default binding on the text widget for <Control-o> adds a newline. This is from the section bindings in the official Tcl/Tk documentation for the text widget:  Control-o opens a new line by inserting a newline character in front of the insertion cursor without moving the insertion cursor.  Returning the string "break" from any binding prevents any further processing of the event. So, you can add a binding on the text widget for control-o that returns the string "break". Since your binding is handled before the default bindings for the widget, this will effectively prevent the default binding from modifying the widget. txt.bind("<Control-o>", lambda e: "break")
Question: We noticed that the mixed usage of fancy indexing and slicing is so confusing and undocumented for multi-dimensional arrays, for example: In [114]: x = np.arange(720).reshape((2,3,4,5,6)) In [115]: x[:,:,:,0,[0,1,2,4,5]].shape Out[115]: (2, 3, 4, 5) In [116]: x[:,:,0,:,[0,1,2,4,5]].shape Out[116]: (5, 2, 3, 5)  I have read the usage of fancy indexing on https://numpy.org/doc/stable/user/basics.indexing.html and I can understand that x[:,0,:,[1,2]] = [x[:,0,:,1], x[:,0,:,2]]. However I cannot understand why the result for above Input [115] and Input [116] differ on the first dimension. Can someone point to where such broadcasting rules are documented? Thanks! I have tried searching the documentation for fancy indexing as well as posting issues to the numpy repo on Github.

Answer: Some additional insight into why there is ambiguity: In the latter case in the question, the 3rd and 5th axes are indexed, and thus disappear from the new array. A new axis (with shape equal to the broadcasting of the indices) has to be added somewhere. If I was numpy, and had to insert a shape (5,) array into the array with "shape" (2, 3, -, 5, -), would I place it in place of the first missing dimension? Or the second? Exactly because a slice separates two advanced indices, numpy can not replace a consecutive set of axes, and thus not know whether to insert the new axis before or after the separating slice(s). As a result, the new axis is inserted at the front: (5, 2, 3, 5) ^ ^^^^^^^--- old dimensions | new dimension  Only in the first case, where the disappearing axes are all adjacent ("shape" (2, 3, 4, -, -)), can the new axes be unambiguously inserted at the end. Note: Behind the scenes numpy always inserts the new axes at the start. It just (mostly for convenience I believe) transposes the array to move the new axes into place when unambiguous. Also interesing is Numpy Enhancement Proposal 21
Question: I am drawing a heatmap and this is my MWE: import matplotlib import seaborn as sns import numpy as np matplotlib.rcParams.update({"figure.dpi": 96}) np.random.seed(7) A = np.random.randint(0,100, size=(20,20)) cmap = matplotlib.cm.get_cmap('viridis').copy() g = sns.heatmap(A, vmin=10, vmax=90, cmap=cmap, cbar_kws={}) # Get the colorbar cbar = g.collections[0].colorbar tick_locations = [*range(15, 86, 10)] # Set the tick positions and labels cbar.set_ticks(tick_locations) cbar.set_ticklabels(tick_locations) plt.show()  This gives me:  But I would like the little horizontal tick marks on the color bar to be white and inside the color bar as in:  How can I do that? (What I am looking for seems to be the default in plotnine/ggplot.)

Answer: add this line before plt.show()... cbar.ax.yaxis.set_ticks_position('both') cbar.ax.tick_params(axis="y",direction="in", color='white')  Output plot
Question: I have a numpy ndarray of shape (5,4,4) that is a set of 5 matrices 4x4. I would like to multiply that ndarray by a matrix of shape (3,5) and I would like to get a numpy ndarray of shape (3,4,4) where each matrix 4x4 in the result is the linear combination of the 5 4x4 matrices with the coefficients coming from the rows of the matrix (3,5). I have the following, very basic, code: import numpy as np np.random.seed(10) X = np.random.rand(5,4,4) A = np.random.rand(3,5) print( A*X )  which generates, a bit as I was expecting, the error 'operands could not be broadcast together with shapes (3,5) (5,4,4)' I was reading a bit about broadcasting but I can't seem to find a way to write the operation in such a way it can be vectorized by Numpy. Has anybody got a similar experience?

Answer: For these type of operations, np.einsum is perfect. Is the following what you want? B = np.einsum('ij,jkl->ikl', A, X) print(B.shape) # (3,4,4)  In words, the string ij,jkl->ikl means:  A has indices (dimensions) i and j respectively X has indices j, k and l respectively ->ikl multiply A[i,j]*X[j,k,l] and sum over j
Question: I have seen other answers on here stating that asyncio doesn't guarantee execution order, only that the order of the outputs will match that of the inputs. Is there a way I can guarantee the execution order For example, if I have a list of the functions I want to run, will calling create task on each of them before calling gather work? Is there something else I can do to achieve this?

Answer: You could do this with asyncio.Event: async def perform_task(listen_event, push_event): await do_some_paralell_work() await listen_event.wait() await do_some_work_in_order() if push_event: await push-event.set() async def main(): events = [asyncio.Event() for i in range(100)] tasks = [asyncio.Task(events[i], events[i+1] if i<100 else None) for i in range(100)] events[0].set() await asyncio.gather(*tasks)  This should let the parallel work run in parallel but then run the final statement in order