eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm new to pandas have tried going through the docs and experiment with various examples, but this problem I'm tacking has really stumped me. I have the following two dataframes (DataA/DataB) which I would like to merge on a per global_index/item/values basis. DataA DataB row item_id valueA row item_id valueB 0 x A1 0 x B1 1 y A2 1 y B2 2 z A3 2 x B3 3 x A4 3 y B4 4 z A5 4 z B5 5 x A6 5 x B6 6 y A7 6 y B7 7 z A8 7 z B8  The list of items(item_ids) is finite and each of the two dataframes represent a the value of a trait (trait A, trait B) for an item at a given global_index value. The global_index could roughly be thought of as a unit of "time" The mapping between each data frame (DataA/DataB) and the global_index is done via the following two mapper DFs: DataA_mapper global_index start_row num_rows 0 0 3 1 3 2 3 5 3 DataB_mapper global_index start_row num_rows 0 0 2 2 2 3 4 5 3  Simply put for a given global_index (eg: 1) the mapper will define a list of rows into the respective DFs (DataA or DataB) that are associated with that global_index. For example, for a global_index value of 0:  In DF DataA rows 0..2 are associated with global_index 0 In DF DataB rows 0..1 are associated with global_index 0  Another example, for a global_index value of 2:  In DF DataB rows 2..4 are associated with global_index 2 In DF DataA there are no rows associated with global_index 2  The ranges [start_row,start_row + num_rows) presented do not overlap each other and represent a unique sequence/range of rows in their respective dataframes (DataA, DataB) In short no row in either DataA or DataB will be found in more than one range.  I would like to merge the DFs so that I get the following dataframe: row global_index item_id valueA valueB 0 0 x A1 B1 1 0 y A2 B2 2 0 z A3 NaN 3 1 x A4 B1 4 1 z A5 NaN 5 2 x A4 B3 6 2 y A2 B4 7 2 z A5 NaN 8 3 x A6 B3 9 3 y A7 B4 10 3 z A8 B5 11 4 x A6 B6 12 4 y A7 B7 13 4 z A8 B8  In the final datafram any pair of global_index/item_id there will ever be either:  a value for both valueA and valueB  a value only for valueA  a value only for valueB  With the requirement being if there is only one value for a given global_index/item (eg: valueA but no valueB) for the last value of the missing one to be used.

Answer: First, you can create the 'global_index' column using the function pd.cut: for df, m in [(df_A, map_A), (df_B, map_B)]: bins = np.insert(m['num_rows'].cumsum().values, 0, 0) # create bins and add zero at the beginning df['global_index'] = pd.cut(df['row'], bins=bins, labels=m['global_index'], right=False)  Next, you can use outer join to merge both data frames: df = df_A.merge(df_B, on=['global_index', 'item_id'], how='outer')  And finally you can use functions groupby and ffill to fill missing values: for val in ['valueA', 'valueB']: df[val] = df.groupby('item_id')[val].ffill()  Output:  item_id global_index valueA valueB 0 x 0 A1 B1 1 y 0 A2 B2 2 z 0 A3 NaN 3 x 1 A4 B1 4 z 1 A5 NaN 5 x 3 A6 B1 6 y 3 A7 B2 7 z 3 A8 NaN 8 x 2 A6 B3 9 y 2 A7 B4 10 z 2 A8 B5 11 x 4 A6 B6 12 y 4 A7 B7 13 z 4 A8 B8
Question: I'm running PyLint on some code and I'm getting the warning of "Either all return statements in a function should return an expression or none of them should. (inconsistent-return-statements)." Here is the code I have: def determine_operand_count(opcode_form, opcode_byte): if opcode_form == OP_FORM.VARIABLE: if opcode_byte & 0b00100000 = 0b00100000: return OP_COUNT.VAR return OP_COUNT.OP2 if opcode_form == OP_FORM.SHORT: if opcode_byte & 0b00110000 == 0b00110000: return OP_COUNT.OP0 return OP_COUNT.OP1 if opcode_form == OP_FORM.LONG: return OP_COUNT.OP2  Here "OP_FORM" and "OP_COUNT" are Enums defined earlier in the code. To me, that code is very readable code and I guess I'm curious what the warning from PyLint is complaining about. In every condition that I have, an "OP_COUNT" type is returned. In fact, if any of these conditionals weren't returning an OP_COUNT, my code would be failing entirely. This seems to be a warning about my "return statements", suggesting that some aren't returning any sort of expression. But that's clearly not true (so far as I can see) as each return statement is returning something. So I'm guessing this has to do with implied returns? But to that point, in my original code I actually held "else" clauses for my internal if statements. But when I did that, PyLint gave me another warning: "Unnecessary 'else' after 'return' (no-else-return)." I did see the following: "How to fix inconsistent return statement in python?", but that didn't seem to reflect the situation in my code. So it's unclear to me how to satisfy PyLint in this case since the code clearly works and seems to be doing what the warning suggests I need to be doing. Given that, I suspect I'm missing something obvious but that I currently lack the intuition for spotting. Any help to get me spotting what I'm missing would be appreciated.

Answer: Pylint complains about what is happening when you reach the very end of the function. What should happen at the end of the function? (added a return and the warning goes away) def determine_operand_count(opcode_form, opcode_byte): if opcode_form == OP_FORM.VARIABLE: if opcode_byte & 0b00100000 == 0b00100000: return OP_COUNT.VAR return OP_COUNT.OP2 if opcode_form == OP_FORM.SHORT: if opcode_byte & 0b00110000 == 0b00110000: return OP_COUNT.OP0 return OP_COUNT.OP1 if opcode_form == OP_FORM.LONG: return OP_COUNT.OP2 return OP_COUNT.UNKNOWN  This code should work, but in my experience is kind of fragile over time because of the indentation. An alternative is to write it as: def determine_operand_count_v2(opcode_form, opcode_byte): def variable_form(opcode_byte): if opcode_byte & 0b00100000 == 0b00100000: return OP_COUNT.VAR return OP_COUNT.OP2 def short_form(opcode_byte): if opcode_byte & 0b00110000 == 0b00110000: return OP_COUNT.OP0 return OP_COUNT.OP1 def long_form(_): return OP_COUNT.OP2 opfcn = {OP_FORM.VARIABLE: variable_form, OP_FORM.SHORT: short_form, OP_FORM.LONG: long_form} return opfcn[opcode_form](opcode_byte)
Question: I wrote this code. It should display the plots in spyder ide. import pandas as pd import numpy as np import matplotlib.pyplot as plt from IPython.display import display from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_absolute_error import scipy.signal import scipy.stats from sklearn.model_selection import train_test_split train = pd.read_csv("train.csv", nrows=9000000,dtype={'acoustic_data':np.int16,'time_to_failure':np.float64}) train.rename({"acoustic_data": "signal", "time_to_failure": "quaketime"}, axis="columns", inplace=True) #visualization(train,title="Acoustic data and time to failure: sampled data") fig, ax = plt.subplots(2,1, figsize=(20,12)) ax[0].plot(train.index.values, train.quaketime.values, c="darkred") ax[0].set_title("Quaketime of 10 Mio rows") ax[0].set_xlabel("Index") ax[0].set_ylabel("Quaketime in ms") ax[1].plot(train.index.values, train.signal.values, c="mediumseagreen") ax[1].set_title("Signal of 10 Mio rows") ax[1].set_xlabel("Index") ax[1].set_ylabel("Acoustic Signal")  Spyder IDE dispaly following message instead of plot  Figures now render in the Plots pane by default. To make them also appear inline in the Console, uncheck "Mute Inline Plotting" under the Plots pane options menu.  I have tested this code in google colab. It plots the required graph. How I can plot in Spyder IDE. I am using Spyder 4.0.1. I did not found any "Mute Inline Plotting" option to uncheck 

Answer: What you have is neither an error nor a warning. It's an instruction. You can find the options following:  Ignore the console outputs, that's just to give you orientation in Spyder 4.0.1.
Question: I'm working with a DataFrame having the following structure: import pandas as pd df = pd.DataFrame({'group':[1,1,1,2,2,2,2,3,3,3], 'brand':['A','B','X','C','D','X','X','E','F','X']}) print(df) group brand 0 1 A 1 1 B 2 1 X 3 2 C 4 2 D 5 2 X 6 2 X 7 3 E 8 3 F 9 3 X  My goal is to view only the groups having exactly one brand X associated to them. Since group number 2 has two observations equal to brand X, it should be filtered out from the resulting DataFrame.  The output should look like this:  group brand 0 1 A 1 1 B 2 1 X 3 3 E 4 3 F 5 3 X  I know I should do a groupby on the group column and then filter those groups having a count of X different than 1. The filtering part is where I struggle. Any help would be appreciated. 

Answer: Use series.eq to check if brand is equal to X , then groupby and transform sum and filter groups in which X count is equal to 1: df[df['brand'].eq('X').groupby(df['group']).transform('sum').eq(1)]    group brand 0 1 A 1 1 B 2 1 X 7 3 E 8 3 F 9 3 X
Question: I'm writing a throwaway script to compute some analytical solutions to a few simulations I'm running.  I would like to implement a function in a way that, based on its inputs, will compute the right answer. So for instance, say I have the following math equation: tmax = (s1 - s2) / 2 = q * (a^2 / (a^2 - b^2))  It seems simple to me that I should be able to do something like: def tmax(s1, s2): return (s1 - s2) / 2 def tmax(a, b, q): return q * (a**2 / (a**2 - b**2))  I may have gotten to used to writing in julia, but I really don't want to complicate this script more than I need to. 

Answer: In statically typed languages like C++, you can overload functions based on the input parameter types (and quantity) but that's not really possible in Python. There can only be one function of any given name. What you can do is to use the default argument feature to select one of two pathways within that function, something like: def tmax(p1, p2, p3 = None): # Two-argument variant has p3 as None. if p3 is None: return (p1 - p2) / 2 # Otherwise, we have three arguments. return (p1 * p1 / (p1 * p1 - p2 * p2)) * p3   If you're wondering why I've change the squaring operations from n ** 2 to n * n, it's because the latter is faster (or it was, at some point in the past, at least for small integral powers like 2 - this is probably still the case but you may want to confirm). A possible case where it may be faster to do g1 ** 2 rather than g1 * g1 is where g1 is a global rather than a local (it takes longer for the Python VM to LOAD_GLOBAL rather than LOAD_FAST). This is not the case with the code posted since the argument is inherently non-global.
Question: I have a list of URLs in a .txt file that I would like to run using selenium. Lets say that the file name is b.txt in it contains 2 urls (precisely formatted as below): https://www.google.com/,https://www.bing.com/, What I am trying to do is to make selenium run both urls (from the .txt file), however it seems that every time the code reaches the "driver.get" line, the code fails. url = open ('b.txt','r') url_rpt = url.read().split(",") options = Options() options.add_argument('--headless') options.add_argument('--disable-gpu') driver = webdriver.Chrome(chrome_options=options) for link in url_rpt: driver.get(link) driver.quit()  The result that I get when I run the code is Traceback (most recent call last): File "C:/Users/ASUS/PycharmProjects/XXXX/Test.py", line 22, in <module> driver.get(link) File "C:\Users\ASUS\AppData\Local\Programs\Python\Python38\lib\site- packages\selenium\webdriver\remote\webdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "C:\Users\ASUS\AppData\Local\Programs\Python\Python38\lib\site- packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\ASUS\AppData\Local\Programs\Python\Python38\lib\site- packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument (Session info: headless chrome=79.0.3945.117)  Any suggestion on how to re-write the code?

Answer: This error message... Traceback (most recent call last): . driver.get(link) . self.execute(Command.GET, {'url': url}) . raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument (Session info: chrome=79.0.3945.117)  ...implies that the url passed as an argument to get() was an argument was invalid. I was able to reproduce the same Traceback when the text file containing the list of urls contains a space character after the seperator of the last url. Possibly a space character was present at the fag end of b.txt as https://www.google.com/,https://www.bing.com/,.  Debugging An ideal debugging approach would be to print the url_rpt which would have revealed the space character as follows:  Code Block: url = open ('url_list.txt','r') url_rpt = url.read().split(",") print(url_rpt)  Console Output: ['https://www.google.com/', 'https://www.bing.com/', ' ']    Solution If you remove the space character from the end your own code would execute just perfecto: options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') url = open ('url_list.txt','r') url_rpt = url.read().split(",") print(url_rpt) for link in url_rpt: driver.get(link) driver.quit()
Question: I have a pandas (version 0.25.3) DataFrame containing a datetime64 column. I'd like to calculate the mean of each column. import numpy as np import pandas as pd n = 1000000 df = pd.DataFrame({ "x": np.random.normal(0.0, 1.0, n), "d": pd.date_range(pd.datetime.today(), periods=n, freq="1H").tolist() })  Calculating the mean of individual columns is pretty much instantaneous. df["x"].mean() ## 1000 loops, best of 3: 1.35 ms per loop df["d"].mean() ## 100 loops, best of 3: 2.91 ms per loop  However, when I use the DataFrame's .mean() method, it takes a really long time. %timeit df.mean() ## 1 loop, best of 3: 9.23 s per loop  It isn't clear to me where the performance penalty comes from.  What is the best way to avoid the slowdown? Should I convert the datetime64 column to a different type? Is using the DataFrame-level .mean() method considered bad form?

Answer: You could restrict it to the numeric values: df.mean(numeric_only=True) Then it runs very fast as well. Here is the text from the documentation:  numeric_only : bool, default None Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.  -- https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html
Question: I am uploading a static site using the Azure Blob storage client library.   blob_service_client = BlobServiceClient.from_connection_string(az_string) blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name) print("\nUploading to Azure Storage as blob:\n\t" + local_file_name) with open('populated.html', "rb") as data: test = blob_client.upload_blob(data, overwrite=True)  This is working but the HTML file is downloading instead of displaying. This is because the content type is wrong: Content-Type: application/octet-stream. Is there any way to set this using upload_blob? Update: To get this working, I needed this:  my_content_settings = ContentSettings(content_type='text/html') blob_client.upload_blob(data, overwrite=True, content_settings=my_content_settings) 

Answer: Looking at the code here, one of the parameters to this method is content_settings which is of type ContentSettings. You can define content_type there.
Question: We are using python API for telegram bots and need to be able to identify the user. Is the chat_id unique for each user connecting the bot?  Can we trust the chat_id to be consistent? e.g same chat_id will tell us that this is the same user, and each user connecting with the bot will have one chat_id that is consistent between sessions?  Thanks 

Answer:  Is the chat_id unique for each user connecting the bot?  Yes chat_id will always be unique for each user connecting to your bot. If the same user sends messages to different bots, they will always 'identify' themselves with their unique id. Keep in mind that getUpdates shows the users id, and the id from the chat. { "ok": true, "result": [ { "update_id": 1234567, "message": { "message_id": 751, "from": { "id": 12122121, <-- user.id "is_bot": false, "first_name": "Me", "last_name": "&", "username": "&&&&", "language_code": "en" }, "chat": { "id": -104235244275, <-- chat_id "title": "Some group", "type": "supergroup" }, "date": 1579999999, "text": "Hi!" } } ] }  According to this post, that chat.id will not change, even if the group is converted to a supergroup Based on comment; small overvieuw of private/group chat example user_1 ---> bot_a in private chat { "message": { "from": { "id": 12345678 <-- id from user_1 }, "chat": { "id": 12345678, <-- send from private chat, so chat is equals to user_id } } } user_2 ---> bot_a in private chat { "message": { "from": { "id": 9876543 <-- id from user_2 }, "chat": { "id": 9876543, <-- send from private chat, so chat is equals to user_id } } } user_1 ---> bot_a in group chat { "message": { "from": { "id": 12345678 <-- id from user_1 }, "chat": { "id": 5646464, <-- send from group chat, so id is from groupchat } } } user_2 ---> bot_a in group chat { "message": { "from": { "id": 9876543 <-- id from user_2 }, "chat": { "id": 5646464, <-- send from group chat, so id is from groupchat } } }
Question: I have a list similar to below ['a','b','c','d','e','f','g','h','i','j']  and I would like to separate by a list of index [1,4]  In this case, it will be  [['a'],['b','c'],['d','e','f','g','h','i','j']]  As [:1] =['a'] [1:4] = ['b','c'] [4:] = ['d','e','f','g','h','i','j']  Case 2: if the list of index is  [0,6]  It will be [[],['a','b','c','d','e'],['f','g','h','i','j']]  As [:0] = [] [0:6] = ['a','b','c','d','e'] [6:] = ['f','g','h','i','j']  Case 3 if the index is [2,5,7]  it will be [['a','b'],['c','d','e'],['h','i','j']] As [:2] =['a','b'] [2:5] = ['c','d','e'] [5:7] = ['f','g'] [7:] = ['h','i','j'] 

Answer: Something along these lines: mylist = ['a','b','c','d','e','f','g','h','i','j'] myindex = [1,4] [mylist[s:e] for s, e in zip([0]+myindex, myindex+[None])]  Output [['a'], ['b', 'c', 'd'], ['e', 'f', 'g', 'h', 'i', 'j']]
Question: I'm creating a small Pandas dataframe: df = pd.DataFrame(data={'colA': [["a", "b", "c"]]})  I take a deepcopy of that df. I'm not using the Pandas method but general Python, right?  import copy df_copy = copy.deepcopy(df)  A df_copy.head() gives the following:   Then I put these values into a dictionary:  mydict = df_copy.to_dict()  That dictionary looks like this:  Finally, I remove one item of the list: mydict['colA'][0].remove("b")  I'm surprized that the values in df_copy are updated. I'm very confused that the values in the original dataframe are updated too! Both dataframes look like this now:  I understand Pandas doesn't really do deepcopy, but this wasn't a Pandas method. My questions are: 1) how can I build a dictionary from a dataframe that doesn't update the dataframe? 2) how can I take a copy of a dataframe which would be completely independent? thanks for your help! Cheers, Nicolas

Answer: TLDR To get deepcopy: df_copy = pd.DataFrame( columns = df.columns, data = copy.deepcopy(df.values) )  Disclaimer  Notice that putting mutable objects inside a DataFrame can be an antipattern so make sure you need it and understand what you are doing.   Why your copy is not independent  When applied on an object, copy.deepcopy is looked up for a _deepcopy_ method of that object, that is called in turn. It's added to avoid copying too much for objects. In the case of a DataFrame instance in version 0.20.0 and above - _deepcopy_ doesn`t work recursively. Similarly, if you will use DataFrame.copy(deep=True) deep copy will copy the data, but will not do so recursively. .   How to solve the problem  To take a truly deep copy of a DataFrame containing a list(or other python objects), so that it will be independent - you can use one of the methods below. df_copy = pd.DataFrame( columns = df.columns, data = copy.deepcopy(df.values) )  For a dictionary, you may use same trick: mydict = pd.DataFrame( columns = df.columns, data = copy.deepcopy(df_copy.values) ).to_dict() mydict['colA'][0].remove("b")  There's also a standard hacky way of deep-copying python objects: import pickle df_copy = pickle.loads(pickle.dumps(df))  Feel free to ask for any clarifications, if needed.
Question: I have a scatterplot that should show the changes in bond lengths depending on temperature. I wanted to give each temperature a specific color, but it doesn't seem to work - plot uses the default seaborn palette. Is there a way to map temperature to color, and make seaborn use it? import pandas as pd import matplotlib.pyplot as plt import seaborn as sns palette = ["#090364", "#091e75", "#093885", "#085396", "#086da6", "#0888b7", "#08a2c7", "#07bdd8", "#07d7e8", "#07f2f9", "#f9ac07", "#c77406", "#963b04", "#640303"] sns.set_style("whitegrid") sns.set_palette(palette) plot = sns.scatterplot(df.loc[:,'length'], df.loc[:,'type'], hue = df.loc[:,'temperature'], legend = False, s = 200) 

Answer: I figured it out. You had to paste the number of colors into the palette: sns.set_style("whitegrid") plot = sns.scatterplot(df.loc[:,'length'], df.loc[:,'type'], hue = df.loc[:,'temperature'], palette=sns.color_palette(palette, len(palette)), legend = False, s = 200)
Question: From the documentation, it says that:  Generic type IO[AnyStr] and its subclasses TextIO(IO[str]) and BinaryIO(IO[bytes]) represent the types of I/O streams such as returned by open(). — Python Docs: typing.IO  The docs did not specify when BinaryIO/TextIO shall be used over their counterparts IO[str] and IO[bytes]. Through a simple inspection of the Python Typeshed source, only 30 hits found when searching for BinaryIO, and 109 hits for IO[bytes]. I was trying to switch to BinaryIO from IO[bytes] for better compatibility with sphinx-autodoc-typehints, but the switch-over has broken many type checks as methods like tempfile.NamedTemporaryFile is typed as IO[bytes] instead of the other. Design-wise speaking, what are the correct situations to use each type of these IO type hints?

Answer: BinaryIO and TextIO directly subclass IO[bytes] and IO[str] respectively, and add on a few extra methods -- see the definitions in typeshed for the specifics. So if you need these extra methods, use BinaryIO/TextIO. Otherwise, it's probably best to use IO[...] for maximum flexibility. For example, if you annotate a method as accepting an IO[str], it's a bit easier for the end-user to provide an instance of that object. Though all this being said, the IO classes in general are kind of messy at present: they define a lot of methods that not all functions will actually need. So, the typeshed maintainers are actually considering breaking up the IO class into smaller Protocols. You could perhaps do the same, if you're so inclined. This approach is mostly useful if you want to define your own IO-like classes but don't want the burden of implementing the full typing.IO[...] API -- or if you're using some class that's almost IO-like, but not quite. All this being said, all three approaches -- using BinaryIO/TextIO, IO[...], or defining more compact custom Protocols -- are perfectly valid. If the sphinx extension doesn't seem to be able to handle one particular approach for some reason, that's probably a bug on their end.