eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Setup:  I'm tring to plot a subplots with plotly library, but can't figure out how to reference a specific subplots' axis to change its' name (or other properties). In Code 1 I show a simple example where I add two plots one on thop of the other with plotly.subplots.make_subplots.  Code 1 import numpy as np from plotly.subplots import make_subplots from math import exp fig = make_subplots(2, 1) x = np.linspace(0, 10, 1000) y = np.array(list(map(lambda x: 1 / (1 + exp(-0.1 * x + 5)), x))) fig.add_trace( go.Scatter( x=x, y=y, name=f'\N{Greek Small Letter Sigma}(x)', showlegend=True ), row=1, col=1 ) x = np.where(np.random.randint(0, 2, 100)==1)[0] fig.add_trace( go.Scatter( x=x, y=np.zeros_like(x), name=f'Plot 2', mode='markers', marker=dict( symbol='circle-open', color='green', size=5 ), showlegend=True ), row=2, col=1 ) fig.show()  What I've Tried I've tried using the fig.update_xaxes() after each trace addition, but it messes the plots and does not produce the desired output, as shown in Code 2. Code 2: import numpy as np from plotly.subplots import make_subplots from math import exp fig = make_subplots(2, 1) x = np.linspace(0, 10, 1000) y = np.array(list(map(lambda x: 1 / (1 + exp(-0.1 * x + 5)), x))) fig.add_trace( go.Scatter( x=x, y=y, name=f'\N{Greek Small Letter Sigma}(x)', showlegend=True ), row=1, col=1 ) fig.update_xaxes(title_text='x') x = np.where(np.random.randint(0, 2, 100)==1)[0] fig.add_trace( go.Scatter( x=x, y=np.zeros_like(x), name=f'Plot 2', mode='markers', marker=dict( symbol='circle-open', color='green', size=5 ), showlegend=True ), row=2, col=1 ) fig.update_xaxes(title_text='active users') fig.show()  which results in (note the active users being printed on the top):  My Questions:  How can I assign the top plot x axis the label x, and active users label to the x axis of the bottom plot? And in general - how can I access the properties of an individual subplot? 

Answer: With the help from this answer I as able to solve it, by referencing the xaxis for plot on the position row=1, col=1 and the xaxis1 for the plot on the row=2, col=1 position. The full solution is in Code 1. Code 1: import numpy as np from plotly.subplots import make_subplots from math import exp fig = make_subplots(2, 1) x = np.linspace(0, 10, 1000) y = np.array(list(map(lambda x: 1 / (1 + exp(-0.1 * x + 5)), x))) fig.add_trace( go.Scatter( x=x, y=y, name=f'\N{Greek Small Letter Sigma}(x)', showlegend=True ), row=1, col=1 ) fig['layout']['xaxis'].update(title_text='x') x = np.where(np.random.randint(0, 2, 100)==1)[0] fig.add_trace( go.Scatter( x=x, y=np.zeros_like(x), name=f'Plot 2', mode='markers', marker=dict( symbol='circle-open', color='green', size=5 ), showlegend=True ), row=2, col=1 ) fig['layout']['xaxis2'].update(title_text='active users') fig.show()  Cheers.
Question: I'm trying to edit the following YAML file db: host: 'x.x.x.x.x' main: password: 'password_main' admin: password: 'password_admin'  To edit the host part, I got it working with sed -i "/^\([[:space:]]*host: \).*/s//\1'$DNS_ENDPOINT'/" config.yml  But I can't find a way to update the password for main and admin (which are different values). I tried to play around with \n and [[:space:]] and got different flavours of: sed -i "/^\([[:space:]]*main:\n*[[:space:]]*password: \).*/s//\1'$DNS_ENDPOINT'/" config.yml  But never got it to work. Any help greatly appreciated! Edit - Requirement: no external binaries/tools. Just good ol' bash.

Answer: $ awk -v new="'sumthin'" 'prev=="main:"{sub(/\047.*/,""); $0=$0 new} {prev=$1} 1' file db: host: 'x.x.x.x.x' main: password: 'sumthin' admin: password: 'password_admin'  or if your new text can contain escape sequences that you don't want expanded (e.g. \t or \n), as seems likely when setting a password, then: new="'sumthin'" awk 'prev=="main:"{sub(/\047.*/,""); $0=$0 ENVIRON["new"]} {prev=$1} 1' file  See How do I use shell variables in an awk script? for why/how I use ENVIRON[] to access a shell variable rather than setting an awk variable in that second script.
Question: I ran the following code a while ago and it worked but now there is the following error. How to solve it?  ValueError: protocol not known.  import json temp = json.dumps([status._json for status in tweet]) # create JSON newdf = pd.read_json(temp, orient='records') 

Answer: As far as I could debug this issue is caused by an update of pandas. The 1.1.0 update had changed few things on the read_json function. I could make my code work when setting pandas version to 1.0.5 https://pandas.pydata.org/docs/whatsnew/v1.1.0.html
Question: Is there a simple way to obtain the hour of the year from a datetime? dt = datetime(2019, 1, 3, 00, 00, 00) # 03/01/2019 00:00 dt_hour = dt.hour_of_year() # should be something like that  Expected output: dt_hour = 48 It would be nice as well to obtain minutes_of_year and seconds_of_year

Answer: One way of implementing this yourself is this: def hour_of_year(dt): beginning_of_year = datetime.datetime(dt.year, 1, 1, tzinfo=dt.tzinfo) return (dt - beginning_of_year).total_seconds() // 3600  This first creates a new datetime object representing the beginning of the year. We then compute the time since the beginning of the year in seconds, divide by 3600 and take the integer part to get the full hours that have passed since the beginning of the year. Note that using the days attribute of the timedelta object will only return the number of full days since the beginning of the year.
Question: I would like to be able to exclude instead of include certain python test files in the pytest.ini configuration file. According to the docs including tests boils down to something like this: # content of pytest.ini [pytest] pytest_files=test_main.py test_common.py  To exclude files however, only command line options are suggested: --ignore=test_common.py How can I actually select files to ignore at the pytest.ini file level?

Answer: You can add any command line options in pytest.ini under addopts. In your case this should work: pytest.ini [pytest] addopts = --ignore=test_common.py  As has been noted in the comments, --ignore takes a path (relative or absolute), not just a module name. From the output of pytest -h:  --ignore=path ignore path during collection (multi-allowed). --ignore-glob=path ignore path pattern during collection (multi-allowed).
Question: I have been trying to parameterize my tests using @pytest.mark.parametrize, and I have a marketer @pytest.mark.test("1234"), I use the value from the test marker to do post the results to JIRA. Note the value given for the marker changes for every test_data. Essentially the code looks something like below. @pytest.mark.foo @pytest.mark.parametrize(("n", "expected"),[ (1, 2), (2, 3)]) def test_increment(n, expected): assert n + 1 == expected  I want to do something like @pytest.mark.foo @pytest.mark.parametrize(("n", "expected"), [ (1, 2,@pytest.mark.test("T1")), (2, 3,@pytest.mark.test("T2")) ])  How to add the marker when using parameterized tests given that the value of the marker is expected to change with each test?

Answer: It's explained here in the documentation: https://docs.pytest.org/en/stable/example/markers.html#marking-individual-tests-when-using-parametrize To show it here as well, it'd be: @pytest.mark.foo @pytest.mark.parametrize(("n", "expected"), [ pytest.param(1, 2, marks=pytest.mark.T1), pytest.param(2, 3, marks=pytest.mark.T2), (4, 5) ])
Question: I am new in python. I want to write an add_function method for this class to add any function to my class dynamically. This function can manipulate some attributes of my class but it is written outside the class. Assume this class and its method. import numpy as np import feature_function as ffun class Features: def __init__(self,x,fs=None): self.x = x self.fs = fs self.label = [] self.features = [] def __mean(self, nan_omit = False, min_samples = 1): out = ffun.mean(self.x, nan_omit = nan_omit, min_samples = min_samples) self.label.extend(['mean']) self.features = np.append(self.features,out)  now I want to write this function outside of the scope of class: def __max(self, nan_omit = False, min_samples = 1): out = ffun.max(self.x, nan_omit = nan_omit, min_samples = min_samples) self.label.extend(['max']) self.features = np.append(self.features,out)  and add it to class. In order to add function to class I have an add_function method inside my class that its arguments are the function that is passed outside the class and also the name of the function.  # Add function dynamically by user def add_function(self, name, methodToRun, type ): name2 = '__' + name # setattr(self, name2, methodToRun) setattr(self, mangle_attr(self, name2), methodToRun)  mangle_attr is a function in order to set private function to a class. def mangle_attr(source, attr): # if source is an object, get the class if not hasattr(source, "__bases__"): source = source.__class__ # mangle attr tmp = {} code = _mangle_template.format(cls=source.__name__, attr=attr) eval(compile(code, '', 'exec'), {}, tmp); return tmp['cls'].mangle.__code__.co_varnames[0]  using setattr doesn't bound __max function to my class, for example __mean is bounded in my class, if I call the function I can see: <bound method Features.__mean of <__main__.Features object at 0x7fd4c91aac10>> however for __max I still see this: <function skewness at 0x7fd2b9dbfb90> do I have a solution for this problem?

Answer: Your addfunction binds its argument to the instance self, not to the class. Bind the argument to the class, e.g. via type(self) or a classmethod: class Features: ... # Add function dynamically by user @classmethod def add_function(cls, name, methodToRun, type ): name2 = '__' + name setattr(cls, mangle_attr(self, name2), methodToRun)  A bound method is a function on the class, looked up through the instance. Adding a function to an instance merely makes it an attribute, which happens to be callable.
Question: I have seen a lot of cases missing values are either filled by mean or medians. I was wondering how can we fill misssing values with frequency. Here is my setup: import numpy as np import pandas as pd df = pd.DataFrame({'sex': [1,1,1,1,0,0,np.nan,np.nan,np.nan]}) df['sex_fillna'] = df['sex'].fillna(df.sex.mode()[0]) print(df) sex sex_fillna 0 1.0 1.0 We have 4 males 1 1.0 1.0 2 1.0 1.0 3 1.0 1.0 4 0.0 0.0 we have 2 females, so ratio is 2 5 0.0 0.0 6 NaN 1.0 Here, I want random choice of [1,1,0] 7 NaN 1.0 eg. 1,1,0 or 1,0,1 or 0,1,1 randomly 8 NaN 1.0  Is there a generic way it can be done so? My attempt df['sex_fillan2'] = df['sex'].fillna(np.random.randint(0,2)) # here the ratio is not guaranteed to approx 4/2 = 2  NOTE This example is only for binary values, I was looking for categorical values having more than two categories. For example: class: A B C 20% 40% 60%  Then instead of filling all nans by class C I would like to fill according to frequency counts. But, is this a good idea? As per some comments, this might or might not be a good idea to impute missing values with different values for different rows, I have created a question in CrossValidated, if you want to give some inputs or see if this is a good idea visit the page: https://stats.stackexchange.com/questions/484467/is-it-better-to-fillnans-based-on-frequency-rather-than-all-values-with-mean-or

Answer: Check with value_counts + np.random.choice s = df.sex.value_counts(normalize=True) df['sex_fillna'] = df['sex'] df.loc[df.sex.isna(), 'sex_fillna'] = np.random.choice(s.index, p=s.values, size=df.sex.isna().sum()) df Out[119]: sex sex_fillna 0 1.0 1.0 1 1.0 1.0 2 1.0 1.0 3 1.0 1.0 4 0.0 0.0 5 0.0 0.0 6 NaN 0.0 7 NaN 1.0 8 NaN 1.0  The output for s index is the category and the value is the probability s Out[120]: 1.0 0.666667 0.0 0.333333 Name: sex, dtype: float64
Question: I've been searching around but couldn't find an easy solution to plot two heatmaps in one graphic by having each square in the heatmap split into two triangles (similar to the attached graphic I saw in a paper). Does anybody know a Python package that is able to do this? I tried seaborn but I don't think it has an easy way to achieve this.  Thank you for your time! -Peter

Answer: plt.tripcolor colors a mesh of triangles similar to how plt.pcolormesh colors a rectangular mesh. Also similar to pcolormesh, care has to be taken that there is one row and one column of vertices less than there are triangles. Furthermore, the arrays need to be made 1D (np.ravel). All this renumbering to 1D can be a bit tricky. As an example, the code below creates a coloring depending on x*y mod 10 and uses two different colormaps for the upper and the lower triangles. import numpy as np import matplotlib.pyplot as plt from matplotlib.tri import Triangulation M = 30 N = 20 x = np.arange(M + 1) y = np.arange(N + 1) xs, ys = np.meshgrid(x, y) zs = (xs * ys) % 10 zs = zs[:-1, :-1].ravel() triangles1 = [(i + j*(M+1), i+1 + j*(M+1), i + (j+1)*(M+1)) for j in range(N) for i in range(M)] triangles2 = [(i+1 + j*(M+1), i+1 + (j+1)*(M+1), i + (j+1)*(M+1)) for j in range(N) for i in range(M)] triang1 = Triangulation(xs.ravel(), ys.ravel(), triangles1) triang2 = Triangulation(xs.ravel(), ys.ravel(), triangles2) img1 = plt.tripcolor(triang1, zs, cmap=plt.get_cmap('inferno', 10), vmax=10) img2 = plt.tripcolor(triang2, zs, cmap=plt.get_cmap('viridis', 10), vmax=10) plt.colorbar(img2, ticks=range(10), pad=-0.05) plt.colorbar(img1, ticks=range(10)) plt.xlim(x[0], x[-1]) plt.ylim(y[0], y[-1]) plt.xticks(x, rotation=90) plt.yticks(y) plt.show()   PS: to have the integer ticks nicely in the center of the cells (instead of at their borders), following changes would be needed: triang1 = Triangulation(xs.ravel()-0.5, ys.ravel()-0.5, triangles1) triang2 = Triangulation(xs.ravel()-0.5, ys.ravel()-0.5, triangles2) # ... plt.xlim(x[0]-0.5, x[-1]-0.5) plt.ylim(y[0]-0.5, y[-1]-0.5) plt.xticks(x[:-1], rotation=90) plt.yticks(y[:-1])
Question: I have an issue with customizing the legend of my plot. I did lot's of customizing but couldnt get my head around this one. I want the symbols (not the labels) to be equally spaced in the legend. As you can see in the example, the space between the circles in the legend, gets smaller as the circles get bigger. any ideas? Also, how can I also add a color bar (in addition to the size), with smaller circles being light red (for example) and bigger circle being blue (for example) here is my code so far: import pandas as pd import matplotlib.pyplot as plt from vega_datasets import data as vega_data gap = pd.read_json(vega_data.gapminder.url) df = gap.loc[gap['year'] == 2000] fig, ax = plt.subplots(1, 1,figsize=[14,12]) ax=ax.scatter(df['life_expect'], df['fertility'], s = df['pop']/100000,alpha=0.7, edgecolor="black",cmap="viridis") plt.xlabel("X") plt.ylabel("Y"); kw = dict(prop="sizes", num=6, color="lightgrey", markeredgecolor='black',markeredgewidth=2) plt.legend(*ax.legend_elements(**kw),bbox_to_anchor=(1, 0),frameon=False, loc="lower left",markerscale=1,ncol=1,borderpad=2,labelspacing=4,handletextpad=2) plt.grid() plt.show()  

Answer: It's a bit tricky, but you could measure the legend elements and reposition them to have a constant inbetween distance. Due to the pixel positioning, the plot can't be resized afterwards. I tested the code inside PyCharm with the 'Qt5Agg' backend. And in a Jupyter notebook, both with %matplotlib inline and with %matplotlib notebook. I'm not sure whether it would work well in all environments. Note that ax.scatter doesn't return an ax (countrary to e.g. sns.scatterplot) but a list of the created scatter dots. import pandas as pd import matplotlib.pyplot as plt from matplotlib.transforms import IdentityTransform from vega_datasets import data as vega_data gap = pd.read_json(vega_data.gapminder.url) df = gap.loc[gap['year'] == 2000] fig, ax = plt.subplots(1, 1, figsize=[14, 12]) fig.subplots_adjust(right=0.8) scat = ax.scatter(df['life_expect'], df['fertility'], s=df['pop'] / 100000, alpha=0.7, edgecolor="black", cmap="viridis") plt.xlabel("X") plt.ylabel("Y") x = 1.1 y = 0.1 is_first = True kw = dict(prop="sizes", num=6, color="lightgrey", markeredgecolor='black', markeredgewidth=2) handles, labels = scat.legend_elements(**kw) inverted_transData = ax.transData.inverted() for handle, label in zip(handles[::-1], labels[::-1]): plt.setp(handle, clip_on=False) for _ in range(1 if is_first else 2): plt.setp(handle, transform=ax.transAxes) if is_first: xd, yd = x, y else: xd, yd = inverted_transData.transform((x, y)) handle.set_xdata([xd]) handle.set_ydata([yd]) ax.add_artist(handle) bbox = handle.get_window_extent(fig.canvas.get_renderer()) y += y - bbox.y0 + 15 # 15 pixels inbetween x = (bbox.x0 + bbox.x1) / 2 if is_first: xd_text, _ = inverted_transData.transform((bbox.x1+10, y)) ax.text(xd_text, yd, label, transform=ax.transAxes, ha='left', va='center') y = bbox.y1 is_first = False plt.show()
Question: I am creating a dictionary object, it gets created while I use "Statement 1", however I get an error message while try create a dictionary object using same keys and values with "Statement 2". Statement 1: dmap = {0: 'Mon', 1: 'Tue', 2: 'Wed', 3: 'Thu', 4: 'Fri', 5: 'Sat', 6: 'Sun'}  Statement 2: dmap = dict(0='Mon', 1='Tue', 2='Wed', 3='Thu', 4='Fri', 5='Sat', 6='Sun'  Error message:  File "<stdin>", line 1 SyntaxError: expression cannot contain assignment, perhaps you meant "=="?  Can someone tell me, why am I allowed to create dictionary with integer keys using Statement 1, but not with Statement 2? Edited Using an updated version of Statement 2, I am able to create dictionary object with below code: dmap = dict(day_0='Mon', day_1='Tue', day_2='Wed', day_3='Thu', day_4='Fri', day_5='Sat', day_6='Sun') 

Answer: dict is a regular callable which accepts keyword arguments. As per the Python syntax, keyword arguments are of the form identifier '=' expression. An identifier may not start with a digit, which excludes number literals.  keyword_item ::= identifier "=" expression  That dict does by default create a dictionary that accepts arbitrary keys does not change the syntax of calls.
Question: On my machine i can't "pip install torch" - i get infamous "single source externally managed error" - i could not fix it and used "conda install torch" from anaconda. Still, checking version is easy - torch.__version__ But how to see where is it installed -the home dir of torch? Suppose if I had had both torches installed via pip and conda - how to know which one is used in a project? import torch print(torch__version__) 

Answer: You can get torch module location which is imported in your script import torch print(torch.__file__)
Question: CODE IN PYCHARM [1]: https://i.sstatic.net/aBP1r.png I tried to make a deep copy of a list l, but seems like the slicing method doesn't work somehow?I don't want the change in x to be reflected in l. So how should I make a deep copy and what is wrong in my code? This was my code- def processed(matrix,r,i): matrix[r].append(i) return matrix l=[[1, 2, 3], [4, 5, 6], [7, 8, 9]] x=l[:] print(processed(x,0,10)) print(l)  OUTPUT- [[1, 2, 3, 10], [4, 5, 6], [7, 8, 9]] [[1, 2, 3, 10], [4, 5, 6], [7, 8, 9]] 

Answer: Your code does indeed succeed in creating a shallow copy. This can be seen by inspecting the IDs of the two outer lists, and noting that they differ. >>> id(l) 140505607684808 >>> id(x) 140505607684680  Or simply comparing using is: >>> x is l False  However, because it is a shallow copy rather than a deep copy, the corresponding elements of the list are the same object as each other: >>> x[0] is l[0] True  This gives you the behaviour that you observed when the sub-lists are appended to. If in fact what you wanted was a deep copy, then you could use copy.deepcopy. In this case the sublists are also new objects, and can be appended to without affecting the originals. >>> from copy import deepcopy >>> l=[[1, 2, 3], [4, 5, 6], [7, 8, 9]] >>> xdeep = deepcopy(l) >>> xdeep == l True >>> xdeep is l False <==== A shallow copy does the same here >>> xdeep[0] is l[0] False <==== But THIS is different from with a shallow copy >>> xdeep[0].append(10) >>> print(l) [[1, 2, 3], [4, 5, 6], [7, 8, 9]] >>> print(xdeep) [[1, 2, 3, 10], [4, 5, 6], [7, 8, 9]]  If you wanted to apply this in your function, you could do: from copy import deepcopy def processed(matrix,r,i): new_matrix = deepcopy(matrix) new_matrix[r].append(i) return new_matrix l = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] x = processed(l,0,10) print(x) print(l)  If in fact you know that the matrix is always exactly 2 deep, then you could do it more efficiently than using deepcopy and without need for the import: def processed(matrix,r,i): new_matrix = [sublist[:] for sublist in matrix] new_matrix[r].append(i) return new_matrix l = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] x = processed(l,0,10) print(x) print(l)
Question: Problem I am trying to deploy a very simple app on Elastic Beanstalk with a small database backend. I try to install mysqlclient as part of the process outlined by AWS here. However, when I deploy my app, I get the following error from my Elastic Beanstalk logs as it tries to download the package: Collecting mysqlclient Using cached mysqlclient-2.0.1.tar.gz (87 kB) 2020/08/21 20:30:16.419082 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed with error exit status 1. Stderr: ERROR: Command errored out with exit status 1: command: /var/app/venv/staging-LQM1lest/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bz45889a/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bz45889a/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1tyle8mv cwd: /tmp/pip-install-bz45889a/mysqlclient/ Complete output (12 lines): /bin/sh: mysql_config: command not found /bin/sh: mariadb_config: command not found /bin/sh: mysql_config: command not found Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-bz45889a/mysqlclient/setup.py", line 15, in <module> metadata, options = get_config() File "/tmp/pip-install-bz45889a/mysqlclient/setup_posix.py", line 65, in get_config libs = mysql_config("libs") File "/tmp/pip-install-bz45889a/mysqlclient/setup_posix.py", line 31, in mysql_config raise OSError("{} not found".format(_mysql_config_path)) OSError: mysql_config not found  Question The mysqlclient is the AWS recommended driver to connect to a MySQL database in RDS using Flask or Django. How do I get it installed with Elastic Beanstalk? Context I've tried to implement my architecture with either (1) a database within the Elastic Beanstalk environment or (2) a regularly deployed RDS instance outside the Elastic Beanstalk environment. In this case, we're going with option (1). I can't install mysql-server or other packages using apt-get as suggested here very easily, which is why I hope this isn't labeled as a duplicate. I don't have access to the underlying servers. I attempted to using platform hooks and .ebextensions, but I wasn't able to get that to work. In this post, I'm trying to step back and see if there is another avenue.

Answer: For Amazon linux 2 you should be able to install it with yum. Thus, you can have in your .ebextensions a file called, e.g. 01_packages.config with the content: packages: yum: MySQL-python: []  You can add further yum dependencies if you require more.