eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to use StackingClassifier to combine some classifiers and then use GridSearchCV to optimize the parameters: clf1 = RandomForestClassifier() clf2 = LogisticRegression() dt = DecisionTreeClassifier() sclf = StackingClassifier(estimators=[clf1, clf2],final_estimator=dt) params = {'randomforestclassifier__n_estimators': [10, 50], 'logisticregression__C': [1,2,3]} grid = GridSearchCV(estimator=sclf, param_grid=params, cv=5) grid.fit(x, y)  But this turns out an error: 'RandomForestClassifier' object has no attribute 'estimators_'  I have used n_estimators. Why it warns me that no estimators_? Usually GridSearchCV is applied to single model so I just need to write the name of parameters of the single model in a dict. I refer to this page https://groups.google.com/d/topic/mlxtend/5GhZNwgmtSg but it uses parameters of early version. Even though I change the newly parameters it doesn't work.  Btw, where can I learn the details of the naming rule of these params?

Answer: First of all, the estimators need to be a list containing the models in tuples with the corresponding assigned names. estimators = [('model1', model()), # model() named model1 by myself ('model2', model2())] # model2() named model2 by myself   Next, you need to use the names as they appear in sclf.get_params(). Also, the name is the same as the one you gave to the specific model in the bove estimators list. So, here for model1 parameters you need:  params = {'model1__n_estimators': [5,10]} # model1__SOME_PARAM  Working toy example: from sklearn.datasets import make_classification from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import StackingClassifier from sklearn.model_selection import GridSearchCV X, y = make_classification(n_samples=1000, n_features=4, n_informative=2, n_redundant=0, random_state=0, shuffle=False) estimators = [('rf', RandomForestClassifier(n_estimators=10, random_state=42)), ('logreg', LogisticRegression())] sclf = StackingClassifier(estimators= estimators , final_estimator=DecisionTreeClassifier()) params = {'rf__n_estimators': [5,10]} grid = GridSearchCV(estimator=sclf, param_grid=params, cv=5) grid.fit(X, y)
Question: I have an app that uses the Spacy model "en_core_web_sm". I have tested the app on my local machine and it works fine. However when I deploy it to Heroku, it gives me this error: "Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory." My requirements file contains spacy==2.2.4. I have been doing some research on this error and found that the model needs to be downloaded separately using this command: python -m spacy download en_core_web_sm I have been looking for ways to add the same to my requirements.txt file but haven't been able to find one that works! I tried this as well - added the below to the requirements file: -e git://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm==2.2.0 but it gave this error: "Cloning git://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz to /app/.heroku/src/en-core-web-sm Running command git clone -q git://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz /app/.heroku/src/en-core-web-sm fatal: remote error: explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz is not a valid repository name" Is there a way to get this Spacy model to load from the requirements file? Or any other fix that is possible? Thank you.

Answer: Ok, so after some more Googling and hunting for a solution, I found this solution that worked: I downloaded the tarball from the url that @tausif shared in his answer, to my local system. Saved it in the directory which had my requirements.txt file. Then I added this line to my requirements.txt file: ./en_core_web_sm-2.2.5.tar.gz Proceeded with deploying to Heroku - it succeeded and the app works perfectly now.
Question: I have the following script that turns a pandas dataframe into an interactive html table with Plotly:  goals_output_df = pd.read_csv(os.path.join(question_dir, "data/output.csv")) fig = go.Figure(data=[go.Table( columnwidth = [15,170,15,35,35], header=dict(values=['<b>' + x + '</b>' for x in list(goals_output_df.columns)], # fill_color='#b9e2ff', line_color='darkslategray', align='center', font=dict(color='black', family="Lato", size=20), height=30 ), cells=dict(values=[goals_output_df[column] for column in goals_output_df.columns], # fill_color='#e6f2fd', line_color='darkslategray', align='left', font=dict(color='black', family="Lato", size=20), height=30 )) ]) fig.update_layout( title="<b>Output summary for %s</b>"%question.strip('question'), font=dict( family="Lato", size=18, color="#000000")) fig.write_html(os.path.join(question_dir, "results/output.html"))  The table contains a column named "Output" which can have one of three values for each row: "YES", "NO" and "BORDERLINE". I want to be able to change the color of the text in the "Output" column such that "YES" is green, "NO" is red and "BORDERLINE" is blue. Any idea how I could do that? Plotly documentation doesn't seem to help.

Answer: It's kind of implicit on this section cell-color-based-on-variable. Here what I'll do in your case. Data import pandas as pd import plotly.graph_objects as go df = pd.DataFrame({"name":["a", "b", "c", "d"], "value":[100,20,30,40], "output":["YES", "NO", "BORDERLINE", "NO"]}) map_color = {"YES":"green", "NO":"red", "BORDERLINE":"blue"} df["color"] = df["output"].map(map_color) cols_to_show = ["name", "value", "output"]  Where cols_to_show are the only columns you want to show in your table. Fill color Here you want to have a standard cell background in all columns but the output one fill_color = [] n = len(df) for col in cols_to_show: if col!='output': fill_color.append(['#e6f2fd']*n) else: fill_color.append(df["color"].to_list())  Table data=[go.Table( # columnwidth = [15,20,30], header=dict(values=[f"<b>{col}</b>" for col in cols_to_show], # fill_color='#b9e2ff', line_color='darkslategray', align='center', font=dict(color='black', family="Lato", size=20), height=30 ), cells=dict(values=df[cols_to_show].values.T, fill_color=fill_color, line_color='darkslategray', align='left', font=dict(color='black', family="Lato", size=20), height=30 )) ] fig = go.Figure(data=data) fig.show()   UPDATE Change text color In case you want to change text color text_color = [] n = len(df) for col in cols_to_show: if col!='output': text_color.append(["black"] * n) else: text_color.append(df["color"].to_list())  and data=[go.Table( # columnwidth = [15,20,30], header=dict(values=[f"<b>{col}</b>" for col in cols_to_show], # fill_color='#b9e2ff', line_color='darkslategray', align='center', font=dict(color='black', family="Lato", size=20), height=30 ), cells=dict(values=df[cols_to_show].values.T, line_color='darkslategray', align='left', font=dict(color=text_color, family="Lato", size=20), height=30 )) ] fig = go.Figure(data=data) fig.show()
Question: While I was running >>> import docker >>> client = docker.from_env() >>> client.containers.list()  I encountered the following error requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))  I think it is because docker-py is not able to get access of the docker daemon. So how do I fix this?

Answer: According to Docker docs you should create a group and attach your user to that group. Create Group sudo groupadd docker  Attach User to Group sudo usermod -aG docker $USER  Reload su -s ${USER}
Question: I am trying to run the simple below snippet  port = int(os.getenv('PORT')) print("Starting app on port %d" % port)  I can understand the PORT is s string but I need to cast to a int. Why I am getting the error TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' 

Answer: You don't have an environment variable called PORT. os.getenv('PORT') -> returns None -> throws exception when you try to convert it to int Before running your script, create in your terminal the environment variable by: export PORT=1234  Or, you can provide a default port in case it's not defined as an environment variable on your machine: DEFAULT_PORT = 1234 port = int(os.getenv('PORT',DEFAULT_PORT)) print("Starting app on port %d" % port)
Question: I am presented with a list made entirely of tuples, such as: lst = [("hello", "Blue"), ("hi", "Red"), ("hey", "Blue"), ("yo", "Green")]  How can I split lst into as many lists as there are colours? In this case, 3 lists [("hello", "Blue"), ("hey", "Blue")] [("hi", "Red")] [("yo", "Green")]  I just need to be able to work with these lists later, so I don't want to just output them to screen.  Details about the list I know that every element of lst is strictly a double-element tuple. The colour is also always going to be that second element of each tuple. The problem Problem is,lst is dependant on user input, so I won't always know how many colours there are in total and what they are. That is why I couldn't predefine variables to store these lists in them.  So how can this be done?

Answer: You could use a collections.defaultdict to group by colour: from collections import defaultdict lst = [("hello", "Blue"), ("hi", "Red"), ("hey", "Blue"), ("yo", "Green")] colours = defaultdict(list) for word, colour in lst: colours[colour].append((word, colour)) print(colours) # defaultdict(<class 'list'>, {'Blue': [('hello', 'Blue'), ('hey', 'Blue')], 'Red': [('hi', 'Red')], 'Green': [('yo', 'Green')]})  Or if you prefer using no libraries, dict.setdefault is an option: colours = {} for word, colour in lst: colours.setdefault(colour, []).append((word, colour)) print(colours) # {'Blue': [('hello', 'Blue'), ('hey', 'Blue')], 'Red': [('hi', 'Red')], 'Green': [('yo', 'Green')]}  If you just want the colour tuples separated into nested lists of tuples, print the values() as a list: print(list(colours.values())) # [[('hello', 'Blue'), ('hey', 'Blue')], [('hi', 'Red')], [('yo', 'Green')]]  Benefit of the above approaches is they automatically initialize empty lists for new keys as you add them, so you don't have to do that yourself.
Question: In pandas, axis=0 represent rows and axis=1 represent columns. Therefore to get the sum of values in each row in pandas, df.sum(axis=0) is called. But it returns a sum of values in each columns and vice-versa. Why???  import pandas as pd df=pd.DataFrame({"x":[1,2,3,4,5],"y":[2,4,6,8,10]}) df.sum(axis=0)  Dataframe:  x y 0 1 2 1 2 4 2 3 6 3 4 8 4 5 10  Output: x 15 y 30  Expected Output: 0 3 1 6 2 9 3 12 4 15 

Answer: I think the right way to interpret the axis parameter is what axis you sum 'over' (or 'across'), rather than the 'direction' the sum is computed in. Specifying axis = 0 computes the sum over the rows, giving you a total for each column; axis = 1 computes the sum across the columns, giving you a total for each row.
Question: I have a BytesIO file-like object, containing a CSV. I want to read it into a Pandas dataframe, without writing to disk in between. MWE In my use case I downloaded the file straight into BytesIO. For this MWE I'll have a file on disk, read it into BytesIO, then read that into Pandas. The disk step is just to make a MWE. file.csv a,b 1,2 3,4  Script: import pandas as pd from io import BytesIO bio = BytesIO() with open('file.csv', 'rb') as f: bio.write(f.read()) # now we have a BytesIO with a CSV df = pd.read_csv(bio)  Result: Traceback (most recent call last): File "pandas-io.py", line 8, in <module> df = pd.read_csv(bio) File "/home/ec2-user/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "/home/ec2-user/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 457, in _read parser = TextFileReader(fp_or_buf, **kwds) File "/home/ec2-user/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 895, in __init__ self._make_engine(self.engine) File "/home/ec2-user/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 1135, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "/home/ec2-user/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 1917, in __init__ self._reader = parsers.TextReader(src, **kwds) File "pandas/_libs/parsers.pyx", line 545, in pandas._libs.parsers.TextReader.__cinit__ pandas.errors.EmptyDataError: No columns to parse from file  Note that this sounds like a similar problem to the title of this post, but the error messages are different, and that post has the X-Y problem.

Answer: The error says the file is empty. That's because after writing to a BytesIO object, the file pointer is at the end of the file, ready to write more. So when Pandas tries to read it, it starts reading after the last byte that was written. So you need to move the pointer back to the start, for Pandas to read. bio.seek(0) df = pd.read_csv(bio)
Question: I am new to learning Python, and I came across colorama. As a test project, I wanted to print out all the available colors in colorama. from colorama import Fore from colorama import init as colorama_init colorama_init(autoreset=True) colors = [x for x in dir(Fore) if x[0] != "_"] for color in colors: print(color + f"{color}")  of course this outputs all black output like this: BLACKBLACK BLUEBLUE CYANCYAN ...  because the Dir(Fore) just gives me a string representation of Fore.BLUE, Fore.GREEN, ... Is there a way to access all the Fore Color property so they actually work, as in: print(Fore.BLUE + "Blue")  Or in other words, this may express my problem better. I wanted to write this: print(Fore.BLACK + 'BLACK') print(Fore.BLUE + 'BLUE') print(Fore.CYAN + 'CYAN') print(Fore.GREEN + 'GREEN') print(Fore.LIGHTBLACK_EX + 'LIGHTBLACK_EX') print(Fore.LIGHTBLUE_EX + 'LIGHTBLUE_EX') print(Fore.LIGHTCYAN_EX + 'LIGHTCYAN_EX') print(Fore.LIGHTGREEN_EX + 'LIGHTGREEN_EX') print(Fore.LIGHTMAGENTA_EX + 'LIGHTMAGENTA_EX') print(Fore.LIGHTRED_EX + 'LIGHTRED_EX') print(Fore.LIGHTWHITE_EX + 'LIGHTWHITE_EX') print(Fore.LIGHTYELLOW_EX + 'LIGHTYELLOW_EX') print(Fore.MAGENTA + 'MAGENTA') print(Fore.RED + 'RED') print(Fore.RESET + 'RESET') print(Fore.WHITE + 'WHITE') print(Fore.YELLOW + 'YELLOW')  in a shorter way: for color in all_the_colors_that_are_available_in_Fore: print('the word color in the representing color') #or something like this? print(Fore.color + color) 

Answer: The reason why it's printing the color name twice is well described in Patrick's comment on the question. Is their a way to access all the Fore Color property so they actualy work as in According to: https://pypi.org/project/colorama/ You can print a colored string using other ways than e.g.print(Fore.RED + 'some red text') You can use colored function from termcolor module which takes a string and a color to colorize that string. But not all Fore colors are supported so you can do the following: from colorama import Fore from colorama import init as colorama_init from termcolor import colored colorama_init(autoreset=True) colors = [x for x in dir(Fore) if x[0] != "_"] colors = [i for i in colors if i not in ["BLACK", "RESET"] and "LIGHT" not in i] for color in colors: print(colored(color, color.lower()))  Hope this answered your question. EDIT: I read more about Fore items and I found that you can retrieve a dictionary containing each color as keys and it's code as values, so you can do the following to include all the colors in Fore: from colorama import Fore from colorama import init as colorama_init colorama_init(autoreset=True) colors = dict(Fore.__dict__.items()) for color in colors.keys(): print(colors[color] + f"{color}")
Question: The fact that NumPy now recommends that new code uses the default_rng() instance instead of numpy.random for new code has got me thinking about how it should be used to yield good results, both performance vice and statistically. This first example is how I first wanted to write: import numpy as np class fancy_name(): def __init__(self): self.rg = np.random.default_rng() self.gamma_shape = 1.0 self.gamma_scale = 1.0 def public_method(self, input): # Do intelligent stuff with input return self.rg.gamma(self.gamma_shape, slef.gamma_scale)  But I have also considered creating a new instance in every function call: import numpy as np class fancy_name(): def __init__(self): self.gamma_shape = 1.0 self.gamma_scale = 1.0 def public_method(self, input): # Do intelligent stuff with input rg = np.random.default_rng() return rg.gamma(self.gamma_shape, slef.gamma_scale)  A third alternative would be to pass the rng as an argument in the function call. This way, the same rng can be used in other parts of the code as well. This is used in a simulation environment that will be called often to sample, for example, transition times. I guess the question is if there are arguments for any of these three methods and if there exists some kind of praxis? Also, any reference to more in-depth explanations of using these random number generators (except for the NumPy doc and Random Sampling article) is of great interest!

Answer: default_rng() isn't a singleton. It makes a new Generator backed by a new instance of the default BitGenerator class. Quoting the docs:  Construct a new Generator with the default BitGenerator (PCG64). ... If seed is not a BitGenerator or a Generator, a new BitGenerator is instantiated. This function does not manage a default global instance.  This can also be tested empirically: In [1]: import numpy In [2]: numpy.random.default_rng() is numpy.random.default_rng() Out[2]: False  This is expensive. You should usually call default_rng() once in your program and pass the generator to anything that needs it. (Yes, this is awkward.)
Question: I want to install Anaconda on my mac (version 10.9.5). The command I used: sh Anaconda3-2020.02-MacOSX-x86_64.sh  Led to this error: Unpacking payload ... Traceback (most recent call last): File "entry_point.py", line 69, in <module> File "concurrent/futures/process.py", line 483, in _chain_from_iterable_of_lists File "concurrent/futures/_base.py", line 598, in result_iterator File "concurrent/futures/_base.py", line 435, in result File "concurrent/futures/_base.py", line 384, in __get_result concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. [1061] Failed to execute script entry_point  After rooting around, I found this suggestion, that I check the hash. I typed this: shasum -a 512 /Users/Slowat/Anaconda3-2020.02-MacOSX-x86_64.sh  and the output was: aa1ed0c40646ba9041abf59c13ce38da1dc51bf15de239b6f966a0b02b4c09c960ae33698c72aa46db41731f8e67938d1972fcb76fa4c5c8081bc0272bb1b535 /Users/Slowat/Anaconda3-2020.02-MacOSX-x86_64.sh  The hash listed here does not match this. So then I thought it was an issue with downloaded, so i deleted the bash script and the directory that anaconda attempted to make during the failed installation. I double checked my python version: localhost:~ Slowat$ python --version Python 3.7.4  and re-downloaded Anaconda3-2020.02-MacOSX-x86_64.sh from here. I re-checked the hash: aa1ed0c40646ba9041abf59c13ce38da1dc51bf15de239b6f966a0b02b4c09c960ae33698c72aa46db41731f8e67938d1972fcb76fa4c5c8081bc0272bb1b535 /Users/Slowat/Anaconda3-2020.02-MacOSX-x86_64.sh  and it's still wrong. Any ideas what I'm doing wrong here (mainly for the original area, checking the hash was just an idea I had).

Answer: I have discovered that conda cannot be run on mac v. 10.9.5; i first updated the mac to 10.11, and then from 10.11 to 10.13 (you have to do it in those two steps). Then conda installed and ran fine.
Question: I am trying to create different subplot and I want for each subplot to have a different background color. I'm using plotly and it makes the thing a little bit more difficult. Any idea? I think the equivalent in matplot is face_color or something like this. fig = make_subplots(rows=1, cols=2) fig.add_trace( go.Scatter( x=list(range(sample_points)), y=data_gx.iloc[scene], name='X-axis', line=dict(color='green', width=2) ), row=1, col=1 ) fig.add_trace( go.Scatter( x=list(range(sample_points)), y=data_ax.iloc[scene], name='X-axis', line=dict(color='green', width=2) ), row=1, col=2 ) 

Answer: fig.add_trace(go.Scatter(x=[-1,2],y= [2,2],fill='tozeroy'),row=1, col=1)  This worked for me. This will draw a line from (-1,2) to (2,2) and will color everything below this points. It can help you color your background of each subplot just by adding in the row and col of subplots. It worked for me, I hope it will work for you too.
Question: I've implemented a python class to generate data as follows:  class Array: def __init__(self): self.arr = [1,2,3,4,5,6,7,8] def __getitem__(self,key): return self.arr[key] a = Array() for i in a: print(i, end = " ")  It runs as expected and I get following  1 2 3 4 5 6 7 8  However, I want to do the same thing for dictionary. Why am I not able to iterate over dictionary like this? class Dictionary: def __init__(self): self.dictionary = {'a' : 1, 'b' : 2, 'c': 3} def __getitem__(self,key): return self.dictionary[key] d = Dictionary() for key in d: print(key, d[key], end=" ")  I expect following output a 1 b 2 c 3  But when I run above code, I get following error :  Traceback (most recent call last): File "<ipython-input-33-4547779db7ec>", line 8, in <module> for key in d: File "<ipython-input-33-4547779db7ec>", line 6, in __getitem__ return self.dictionary[key] KeyError: 0  We can iterate over normal dictionary like this : for key in d. This iterates over all the keys, is it not possible to iterate like this using __getitem__()?

Answer: A for loop works with iterators, objects you can pass to next. An object is an iterator if it has a __next__ method. Neither of your classes does, so Python will first pass your object to iter to get an iterator. The first thing iter tries to do is call the object's __iter__ method. Neither of your classes defines __iter__, either, so iter next checks if its object defines __getitem__. Both of your classes do, so iter returns an object of type iterator, whose __next__ method can be imagined to be something like def __next__(self): try: rv = self.thing.__getitem__(self.i) except IndexError: raise StopIteration self.i += 1 return rv  (The iterator holds a reference to the thing which defined __getitem__, as well as the value of i to track state between calls to __next__. i is presumed to be initialized to 0.) For Array, this works, because it has integer indices. For Dictionary, though, 0 is not a key, and instead of raising an IndexError, you get a KeyError with the __next__ method does not catch. (This is alluded to in the documentation for __getitem__:  Note for loops expect that an IndexError will be raised for illegal indexes to allow proper detection of the end of the sequence.  ) To make your Dictionary class iterable, define __iter__ class Dictionary: def __init__(self): self.dictionary = {'a' : 1, 'b' : 2, 'c': 3} def __getitem__(self,key): return self.dictionary[key] def __iter__(self): return iter(self.dictionary)  dict.__iter__ returns a value of type dict_keyiterator, which is the thing that yields the dict's keys, which you can use with Dictionary.__getitem__.
Question: I would like to make the following sum given two lists: a = [0,1,2,3,4,5,6,7,8,9] b = [2,3,5]  The result should be the sum of the every b element of a like:  b[0] = 2 so the first sum result should be: sum(a[0:2]) b[1] = 3 so the second sum result should be: sum(a[2:5]) b[2] = 5 so the third sum result should be: sum(a[5:10])  The printed result: 1,9,35

Answer: You can make use of np.bincount with weights: groups = np.repeat(np.arange(len(b)), b) np.bincount(groups, weights=a)  Output: array([ 1., 9., 35.])