eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Hi I need to install from a branch of a git repo. I want to include it on the requirements.txt so that it would install using the command pip install -r requirements.txt What I know is how to install from master branch (See git ssh entry below): This is my requirements.txt networkx==2.4 numpy==1.18.1 opencv-python==4.2.0.32 scipy==1.4.1 git+ssh://git@gitlab.com/project/project-utils.git  What if I want to install from a specific branch namely 1-fix-test on ssh://git@gitlab.com/project/project-utils.git. How do I include the branch name with the ssh address?

Answer: According to the document, you can add branch name or commit hash after @: git+ssh://git@gitlab.com/project/project-utils.git@1-fix-test
Question: I heard from one guy that you should not use magic methods directly. and I think in some use cases I would have to use magic methods directly. So experienced devs, should I use python magic methods directly?

Answer: I intended to show some benefits of not using magic methods directly: 1- Readability: Using built-in functions like len() is much more readable than its relevant magic/special method __len__(). Imagine a source code full of only magic methods instead of built-in function... thousands of underscores...  2- Comparison operators: class C: def __lt__(self, other): print('__lt__ called') class D: pass c = C() d = D() d > c d.__gt__(c)  I haven't implemented __gt__ for neither of those classes, but in d > c when Python sees that class D doesn't have __gt__, it checks to see if class C implements __lt__. It does, so we get '__lt__ called' in output which isn't the case with d.__gt__(c).  3- Extra checks: class C: def __len__(self): return 'boo' obj = C() print(obj.__len__()) # fine print(len(obj)) # error  or: class C: def __str__(self): return 10 obj = C() print(obj.__str__()) # fine print(str(obj)) # error  As you see, when Python calls that magic methods implicitly, it does some extra checks as well.  4- This is the least important but using let's say len() on built-in data types such as str gives a little bit of speed as compared to __len__(): from timeit import timeit string = 'abcdefghijklmn' print(timeit("len(string)", globals=globals(), number=10_000_000)) print(timeit("string.__len__()", globals=globals(), number=10_000_000))  output: 0.5442426 0.8312854999999999  It's because of the lookup process(__len__ in the namespace), If you create a bound method before timing, it's gonna be faster. bound_method = string.__len__ print(timeit("bound_method()", globals=globals(), number=10_000_000))
Question: Let's say I have the following array:  a = [4,2,3,1,4]  Then I sort it: b = sorted(A) = [1,2,3,4,4]  How could I have a list that map where each number was, ex: position(b,a) = [3,1,2,0,4]  to clarify this list contains the positions not values) (ps' also taking in account that first 4 was in position 0)

Answer: b = sorted(enumerate(a), key=lambda i: i[1])  This results is a list of tuples, the first item of which is the original index and second of which is the value: [(3, 1), (1, 2), (2, 3), (0, 4), (4, 4)]
Question: I have set up couple of environments with Data Science libraries like pandas, numpy, matplotlib, scikit-learn, tensorflow etc.. However I cannot update some packages to the latest version. E.g. conda update pandas  will tell me I have the latest version available however I know for sure the latest version is 1.+ (mine is 0.25) Is there a way to see which packages prevent a specific package from updating?

Answer: There is a way to do it using the drop-in replacement mamba. All you have to do is provide the version of the package you want to update to, and mamba will tell you what's preventing it from updating. E.g., in my case, I wanted to update snakemake to version > 7. But mamba update snakemake only gave me 6.15. So I ran: mamba install snakemake=7, and the result was informative: Looking for: ['snakemake=7'] Pinned packages: - python 3.8.* - bcbio-gff 0.6.7.* Encountered problems while solving: - nothing provides yte >=1.0,<2.0 needed by snakemake-minimal-7.0.0-pyhdfd78af_0  It turns out I had forgotten to include -c conda-forge which is where yte was to come from.
Question: My pytest unit test keeps returning the error ModuleNotFoundError: No module name billing. Oddly enough the send_invoices method in the billing module is able to be called when I remove the patch statement. Why is mock.patch unable to find the billing module and patch the method if this is the case? billing.py  import pdfkit from django.template.loader import render_to_string from django.core.mail import EmailMessage from projectxapp.models import User Class Billing: #creates a pdf of invoice. Takes an invoice dictionary def create_invoice_pdf(self, invoice, user_id): #get the html of the invoice file_path ='/{}-{}.pdf'.format(user_id, invoice['billing_period']) invoice_html = render_to_string('templates/invoice.html', invoice) pdf = pdfkit.from_file(invoice_html, file_path) return file_path, pdf #sends invoice to customer def send_invoices(self, invoices): for user_id, invoice in invoices.items(): #get customers email email = User.objects.get(id=user_id).email billing_period = invoice['billing_period'] invoice_pdf = self.create_invoice_pdf(invoice, user_id) email = EmailMessage( 'Statement of credit: {}-{}'.format(user_id, billing_period), 'Attached is your statement of credit.\ This may also be viewed on your dashboard when you login.', 'no-reply@trevoe.com', [email], ).attach(invoice_pdf) email.send(fail_silently=False) return True  test.py from mock import patch from projectxapp import billing @pytest.mark.django_db def test_send_invoice(): invoices = { 1: { 'transaction_processing_fee': 2.00, 'service_fee': 10.00, 'billing_period': '2020-01-02' }, 2: { 'transaction_processing_fee': 4.00, 'service_fee': 20.00, 'billing_period': '2020-01-02' } } with patch('services.billing.Billing().create_invoice_pdf') as p1: p1.return_value = '/invoice.pdf' test = billing.Billing().send_invoices(invoices) assert test == True 

Answer: Solution Since I had already imported the module of the method I needed to patch. I didn't need to use the full path including the package name. Changed patch('projectxapp.billing.Billing.create_invoice_pdf') to this patch('billing.Billing.create_invoice_pdf') From the unittest documentation:  target should be a string in the form 'package.module.ClassName'. The target is imported and the specified object replaced with the new object, so the target must be importable from the environment you are calling patch() from. The target is imported when the decorated function is executed, not at decoration time.
Question: Python's poetry dependency manager allows specifying optional dependencies via command: $ poetry add --optional redis  Which results in this configuration: [tool.poetry.dependencies] python = "^3.8" redis = {version="^3.4.1", optional=true}  However how do you actually install them? Docs seem to hint to: $ poetry install -E redis  but that just throws and error: Installing dependencies from lock file [ValueError] Extra [redis] is not specified. 

Answer: You need to add a tool.poetry.extras group to your pyproject.toml if you want to use the -E flag during install, as described in this section of the docs: [tool.poetry.extras] caching = ["redis"]   The key refers to the word that you use with poetry install -E, and the value is a list of packages that were marked as --optional when they were added. There currently is no support for making optional packages part of a specific group during their addition, so you have to maintain this section in your pyproject.toml file by hand. The reason behind this additional layer of abstraction is that extra-installs usually refer to some optional functionality (in this case caching) that is enabled through the installation of one or more dependencies (in this case just redis). poetry simply mimics setuptools' definition of extra-installs here, which might explain why it's so sparingly documented.
Question: New mackbookpro running Catalina. Installed anaconda using homebrew. Tried to install psycopg2 using the command conda install -c anaconda psycopg2 but failed due to package conflicts. Here's some of the output from the attempted install: $ conda install -c anaconda psycopg2 Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: / Found conflicts! Looking for incompatible packages. This can take several minutes. Press CTRL-C to abort. failed \ UnsatisfiableError: The following specifications were found to be incompatible with each other: Output in format: Requested package -> Available versions Package six conflicts for: pyopenssl -> cryptography[version='>=2.8'] -> six[version='>=1.4.1'] pytest-doctestplus -> pytest[version='>=3.0'] -> six[version='>=1.10.0'] python-dateutil -> six[version='>=1.5'] nltk -> six  Any thoughts on what's happening or how to fix?

Answer: The reason might be there are too many conflicts between [anaconda==2020.02] and [70+ PACKAGES] Try the following worked for me: conda -V conda update -n base conda To ensure you are in version conda 4.8.2 Then conda update --all Then the following packages will be DOWNGRADED: anaconda 2020.02-py37_0 --> custom-py37_1 Then conda install psycopg2 Then packages libpq,psycopg2 would be installed, some packages would be updated, some packages would be superseded by higher-priority channel.
Question: I would need to take the question about json serialization of @dataclass from Make the Python json encoder support Python's new dataclasses a bit further: consider when they are in a nested structure.  Consider: import json from attr import dataclass from dataclasses_json import dataclass_json @dataclass @dataclass_json class Prod: id: int name: str price: float prods = [Prod(1,'A',25.3),Prod(2,'B',79.95)] pjson = json.dumps(prods)  That gives us: TypeError: Object of type Prod is not JSON serializable  Note the above does incorporate one of the answers https://stackoverflow.com/a/59688140/1056563 . It claims to support the nested case via the dataclass_json decorator . Apparently that does not actually work.  I also tried another of the answers https://stackoverflow.com/a/51286749/1056563 : class EnhancedJSONEncoder(json.JSONEncoder): def default(s, o): if dataclasses.is_dataclass(o): return dataclasses.asdict(o) return super().default(o)  And I created a helper method for it: def jdump(s,foo): return json.dumps(foo, cls=s.c.EnhancedJSONEncoder)  But using that method also did not effect the (error) result. Any further tips?

Answer: You can use a pydantic library. From the example in documentation from pydantic import BaseModel class BarModel(BaseModel): whatever: int class FooBarModel(BaseModel): banana: float foo: str bar: BarModel m = FooBarModel(banana=3.14, foo='hello', bar={'whatever': 123}) # returns a dictionary: print(m.dict()) """ { 'banana': 3.14, 'foo': 'hello', 'bar': {'whatever': 123}, } """ print(m.dict(include={'foo', 'bar'})) #> {'foo': 'hello', 'bar': {'whatever': 123}} print(m.dict(exclude={'foo', 'bar'})) #> {'banana': 3.14}
Question: I'm mainly working on PyCharm, and lot of times, I come into a situation where it would be much better if I can collapse/fold all method bodies, and leave their names only. The picture below is the the result I want, but I can't find the shortcut to do this thing. If you know any, let me know. 

Answer: You can collapse the code as shown in the screenshot going to Code > Folding > Expand All to Level > 1 or using the keyboard shortcut Ctrl + Shift + NumPad *, 1. The level is absolute relative to the module level. If you have nested methods or classes you can select them individually an use the equivalent Expand to Level instead of Expand All to Level.
Question: I am trying to make a fully connected model using tensorflow.keras, here is my code from tensorflow.keras.models import Model from tensorflow.keras.layers import Input, Dense, Flatten def load_model(input_shape): input = Input(shape = input_shape) dense_shape = input_shape[0] x = Flatten()(input) x = Dense(dense_shape, activation='relu')(x) x = Dense(dense_shape, activation='relu')(x) x = Dense(dense_shape, activation='relu')(x) x = Dense(dense_shape, activation='relu')(x) x = Dense(dense_shape, activation='relu')(x) output = Dense(10, activation='softmax') model = Model(input , output) model.summary() return model  but when I call the model model = load_model((120,))  I have this error 'Dense' object has no attribute 'op'  How can I fix this?

Answer: You are missing (x) after your output layer. Try output = Dense(10 , activation = 'softmax')(x)
Question: When I use Plotly express to plot different parameters with different ranges - in the example below, BloodPressureHigh, Height(cm), Weight(kg), and BloodPressureLow - using the facet_col argument, I am unable to get the resulting plot to display the unique YTicks for each of the faceted plots. Is there an easy method for the fig object to show each set of YTicks in the resulting faceted plot? Otherwise, as you can see in the resulting image, it is unclear that each box plot is on its own unique YAxis. import plotly.express as px import pandas as pd temp = [ {"Clinic": "A", "Subject": "Bill", "Height(cm)": 182, "Weight(kg)": 101, "BloodPressureHigh": 128, "BloodPressureLow": 90}, {"Clinic": "A", "Subject": "Susie", "Height(cm)": 142, "Weight(kg)": 67, "BloodPressureHigh": 120, "BloodPressureLow": 70}, {"Clinic": "B", "Subject": "John", "Height(cm)": 202, "Weight(kg)": 89, "BloodPressureHigh": 118, "BloodPressureLow": 85}, {"Clinic": "B", "Subject": "Stacy", "Height(cm)": 156, "Weight(kg)": 78, "BloodPressureHigh": 114, "BloodPressureLow": 76}, {"Clinic": "B", "Subject": "Lisa", "Height(cm)": 164, "Weight(kg)": 59, "BloodPressureHigh": 112, "BloodPressureLow": 74} ] df = pd.DataFrame(temp) # Melt the dataframe so I can use plotly express to plot distributions of all variables df_melted = df.melt(id_vars=["Clinic", "Subject"]) # Plot distributions, with different parameters in different columns fig = px.box(df_melted, x="Clinic", y="value", facet_col="variable", boxmode="overlay" ) # Update the YAxes so that the faceted column plots no longer share common YLimits fig.update_yaxes(matches=None) # Last step needed: Add tick labels to each yaxis so that the difference in YLimits is clear?  

Answer: Does this help you? fig = px.box(df_melted, x="Clinic", y="value", facet_col="variable", boxmode="overlay") fig.update_yaxes(matches=None) fig.for_each_yaxis(lambda yaxis: yaxis.update(showticklabels=True)) fig.show()
Question: I'm getting Arrow Key values in my Python Input using input(). This only happens during the time of execution of a Python Script. It doesn't happen if Input is taken from the Interpreter. The Arrow Key values I'm referring to: Why does the terminal show "^[[A" "^[[B" "^[[C" "^[[D" when pressing the arrow keys in Ubuntu? Contents of Script File: s = input("Enter Something: ") print(s)  Terminal Output: $ python input_example.py Enter Something: Now Pressing Left Arrow Key^[[D^[[D^[[D^[[D Now Pressing Left Arrow Key  I'm unable to navigate (or say change cursor position) left or right while writing input cause the Arrow Key Values Show up in the Input. Is there any way to avoid them? In Terminal, generally, one can change the cursor position, this problem doesn't happen unlike Python's input(). P.s. I don't want to change any settings in bash cause I'm trying to write a script that works in all consoles. I'm a noob so I don't understand many things. I hope this community can help me out.

Answer: Found a way to prevent this! You just have to import the readline module import readline  This will make the standard input() method utilize some of its utilities, enabling normal arrow-key usage and more.
Question: I have a 3 columns in my dataframe. I have charted them all in plotly, and the below code puts them side by side in a subplot. I would like to change the third chart 'c' to have a logarithmic scale. Is this possible? fig = make_subplots(rows=1, cols=3) fig.add_trace( go.Scatter(x = df.index,y = df['a'],mode = 'lines+markers',name = 'Daily') fig.add_trace( go.Scatter(x = df.index,y = df['b'],mode = 'lines+markers',name = 'Total' ) fig.add_trace( go.Scatter(x = df.index,y = df['c'],mode = 'lines+markers',name = 'Total' ) fig.layout() fig.update_layout(height=600, width=800, title_text="Subplots") fig.show() 

Answer: See the section on "Customizing Subplot Axes" in the Plotly documentation. I included an example below for 2 subplots, but the logic is the same regardless of the number of subplots. import plotly.graph_objects as go from plotly.subplots import make_subplots fig = make_subplots(rows=1, cols=2, subplot_titles=("Default Scale", "Logarithmic Scale")) # subplot in default scale fig.add_trace(go.Scatter(x=[0.1, 0.2, 0.3, 0.4, 0.5], y=[1.105, 1.221, 1.35, 1.492, 1.649]), row=1, col=1) fig.update_xaxes(title_text="x-axis in default scale", row=1, col=1) fig.update_yaxes(title_text="y-axis in default scale", row=1, col=1) # subplot in logarithmic scale fig.add_trace(go.Scatter(x=[0.1, 0.2, 0.3, 0.4, 0.5], y=[1.105, 1.221, 1.35, 1.492, 1.649]), row=1, col=2) fig.update_xaxes(title_text="x-axis in logarithmic scale", type="log", row=1, col=2) fig.update_yaxes(title_text="y-axis in logarithmic scale", type="log", row=1, col=2) fig.update_layout(showlegend=False) fig.show()
Question: I am new to using CloudFormation / CDK and am having trouble figuring out to deploy my stacks without error. Currently I am using the python CDK to create a bucket. This bucket will hold model files and I need to ensure that the bucket deployed in this stack retains data over time / new deployments. From my initial tests, it seems that if bucket_name is not specified, the CDK will randomly generate a new bucket name on deployment, which is not ideal.  Here is the snippet used to create the bucket: bucket = aws_s3.Bucket(self, "smartsearch-bucket", bucket_name= 'mybucketname') The first time I run cdk deploy, there are no problems and the bucket is created. The second time I run cdk deploy, I get an error stating that my S3 bucket already exists. What else is needed so that I can redeploy my stack using a predetermined S3 bucket name?

Answer: I ran into the same issue, and it was due to the reason that bucket was already created by me manually earlier for some testing, NOT by ECS stack initially. Deleting the bucket definitely makes ECS deployment to work fine, as it did for you and I tested this running the deployment multiple times. Ensure that no ECS resources are pre-created manually. The way ECS would identify if it has to re-create a resource is via these tags:
Question: i want to play with the InstagramAPI and write some code for like getting a list of my follower and something like that. I am really new to that topic.  What is the best way to do this? Is there a Python-Lib for handle those json request or should I send them directly to the (new? graphAPI, displayAPI) InstagramAPI?  Appreciate every advice I can get. Thanks :)

Answer: LevPasha's Instagram-API-python, instabot, and many other API's are no longer functional as of Oct 24, 2020 after Facebook deprecated the legacy API and now has a new, authentication-required, API. It now requires registering your app with Facebook to be able to get access to many of the API features (via oembed) that were previously available without any authentication. See https://developers.facebook.com/docs/instagram/oembed/ for more details on the new implementation and how to migrate. You should still be able to get a list of your followers, etc. via the new oEmbed API and python--it will require registering the app, making a call to the new GET API with your authentication key via the python requests package, and then processing the result.
Question: I'm trying to deploy a Streamlit app written in python to Heroku. My whole directory is 4.73 MB, where 4.68 MB is my ML model. My requirements.txt looks like this: absl-py==0.9.0 altair==4.0.1 astor==0.8.1 attrs==19.3.0 backcall==0.1.0 base58==2.0.0 bleach==3.1.3 blinker==1.4 boto3==1.12.29 botocore==1.15.29 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 click==7.1.1 colorama==0.4.3 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0 docutils==0.15.2 entrypoints==0.3 enum-compat==0.0.3 future==0.18.2 gast==0.2.2 google-auth==1.11.3 google-auth-oauthlib==0.4.1 google-pasta==0.2.0 grpcio==1.27.2 h5py==2.10.0 idna==2.9 importlib-metadata==1.5.2 ipykernel==5.2.0 ipython==7.13.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.16.0 Jinja2==2.11.1 jmespath==0.9.5 joblib==0.14.1 jsonschema==3.2.0 jupyter-client==6.1.1 jupyter-core==4.6.3 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.1.0 Markdown==3.2.1 MarkupSafe==1.1.1 matplotlib==3.2.1 mistune==0.8.4 nbconvert==5.6.1 nbformat==5.0.4 notebook==6.0.3 numpy==1.18.2 oauthlib==3.1.0 opencv-python==4.2.0.32 opt-einsum==3.2.0 pandas==1.0.3 pandocfilters==1.4.2 parso==0.6.2 pathtools==0.1.2 pickleshare==0.7.5 Pillow==7.0.0 prometheus-client==0.7.1 prompt-toolkit==3.0.4 protobuf==3.11.3 pyasn1==0.4.8 pyasn1-modules==0.2.8 pydeck==0.3.0b2 Pygments==2.6.1 pyparsing==2.4.6 pyrsistent==0.16.0 python-dateutil==2.8.0 pytz==2019.3 pywinpty==0.5.7 pyzmq==19.0.0 requests==2.23.0 requests-oauthlib==1.3.0 rsa==4.0 s3transfer==0.3.3 scikit-learn==0.22.2.post1 scipy==1.4.1 Send2Trash==1.5.0 six==1.14.0 sklearn==0.0 streamlit==0.56.0 tensorboard==2.1.1 tensorflow==2.1.0 tensorflow-estimator==2.1.0 termcolor==1.1.0 terminado==0.8.3 testpath==0.4.4 toml==0.10.0 toolz==0.10.0 tornado==5.1.1 traitlets==4.3.3 tzlocal==2.0.0 urllib3==1.25.8 validators==0.14.2 watchdog==0.10.2 wcwidth==0.1.9 webencodings==0.5.1 Werkzeug==1.0.0 widgetsnbextension==3.5.1 wincertstore==0.2 wrapt==1.12.1 zipp==3.1.0  When I push my app to Heroku, the message is: remote: -----> Discovering process types remote: Procfile declares types -> web remote: remote: -----> Compressing... remote: ! Compiled slug size: 623.5M is too large (max is 500M). remote: ! See: http://devcenter.heroku.com/articles/slug-size remote: remote: ! Push failed  How can my slug size be too large? Is it the size of the requirements? Then how is it possible to deploy a python app using tensorflow to Heroku after all? Thanks for the help!

Answer: I have already answered this here. Turns out the Tensorflow 2.0 module is very large (more than 500MB, the limit for Heroku) because of its GPU support. Since Heroku doesn't support GPU, it doesn't make sense to install the module with GPU support. Solution: Simply replace tensorflow with tensorflow-cpu in your requirements. This worked for me, hope it works for you too!
Question: Pandas throws a Future Warning when I apply a function to multiple columns of a groupby object. It suggests to use a list as index instead of tuples. How would one go about this? >>> df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]]) >>> df.groupby([0,1])[1,2].apply(sum) <stdin>:1: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead. 1 2 0 1 1 2 2 3 4 5 5 6 7 8 8 9 

Answer: This warning was introduced in pandas 1.0.0, following a discussion on GitHub. So best use what was suggested there: df.groupby([0, 1])[[1, 2]].apply(sum)  It's also possible to move the slicing operation to the end, but that is not as efficient: df.groupby([0, 1]).apply(sum).loc[:, 1:]  Thanks @ALollz and @cmosig for helpful comments.