eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: At the time of asking this question, I'm using Python 3.8 When I say efficient, I'm only referring to the speed at which the strings are concatenated, or in more technical terms: I'm asking about the time complexity, not accounting the space complexity. The only methods I can think of at the moment are the following 3 given that: a = 'start' b = ' end'  Method 1 result = a + b Method 2 result = ''.join((a, b)) Method 3 result = '{0}{1}'.format(a, b) I want to know which of these methods are faster, or if there are other methods that are more efficient. Also, if you know if either of these methods performs differently with more strings or longer strings, please include that in your answer. Edit After seeing all the comments and answers, I have learned a couple of new ways to concatenate strings, and I have also learned about the timeit library. I will report my personal findings below: >>> import timeit >>> print(timeit.Timer('result = a + b', setup='a = "start"; b = " end"').timeit(number=10000)) 0.0005306000000473432 >>> print(timeit.Timer('result = "".join((a, b))', setup='a = "start"; b = " end"').timeit(number=10000)) 0.0011297000000354274 >>> print(timeit.Timer('result = "{0}{1}".format(a, b)', setup='a = "start"; b = " end"').timeit(number=10000)) 0.002327799999989111 >>> print(timeit.Timer('result = f"{a}{b}"', setup='a = "start"; b = " end"').timeit(number=10000)) 0.0005772000000092703 >>> print(timeit.Timer('result = "%s%s" % (a, b)', setup='a = "start"; b = " end"').timeit(number=10000)) 0.0017815999999584164  It seems that for these small strings, the traditional a + b method is the fastest for string concatenation. Thanks for all of the answers!

Answer: Let's try it out! We can use timeit.timeit() to run a statement many times and return the overall duration. Here, we use s to setup the variables a and b (not included in the overall time), and then run the various options 10 million times. >>> from timeit import timeit >>> >>> n = 10 * 1000 * 1000 >>> s = "a = 'start'; b = ' end'" >>> >>> timeit("c = a + b", setup=s, number=n) 0.4452877212315798 >>> >>> timeit("c = f'{a}{b}'", setup=s, number=n) 0.5252049304544926 >>> >>> timeit("c = '%s%s'.format(a, b)", setup=s, number=n) 0.6849184390157461 >>>> >>> timeit("c = ''.join((a, b))", setup=s, number=n) 0.8546998891979456 >>> >>> timeit("c = '%s%s' % (a, b)", setup=s, number=n) 1.1699129864573479 >>> >>> timeit("c = '{0}{1}'.format(a, b)", setup=s, number=n) 1.5954962372779846  This shows that unless your application's bottleneck is string concatenation, it's probably not worth being too concerned about...  The best case is ~0.45 seconds for 10 million iterations, or about 45ns per operation. The worst case is ~1.59 seconds for 10 million iterations, or about 159ns per operation.  Depending on the performance of your system, you might see a speed improvement in the order of a few seconds if you're performing literally millions of operations. Note that your results may vary quite drastically depending on the lengths (and number) of the strings you're concatenating, and the hardware you're running on.
Question: I have always used: r = requests.get(url) if r.status_code == 200: # my passing code else: # anything else, if this even exists  Now I was working on another issue and decided to allow for other errors and am instead now using: try: r = requests.get(url) r.raise_for_status() except requests.exceptions.ConnectionError as err: # eg, no internet raise SystemExit(err) except requests.exceptions.HTTPError as err: # eg, url, server and other errors raise SystemExit(err) # the rest of my code is going here  With the exception that various other errors could be tested for at this level, is one method any better than the other?

Answer: Response.raise_for_status() is just a built-in method for checking status codes and does essentially the same thing as your first example. There is no "better" here, just about personal preference with flow control. My preference is toward try/except blocks for catching errors in any call, as this informs the future programmer that these conditions are some sort of error. If/else doesn't necessarily indicate an error when scanning code.  Edit: Here's my quick-and-dirty pattern. import time from http import HTTPStatus import requests from requests.exceptions import HTTPError url = "https://theurl.com" retries = 3 retry_codes = [ HTTPStatus.TOO_MANY_REQUESTS, HTTPStatus.INTERNAL_SERVER_ERROR, HTTPStatus.BAD_GATEWAY, HTTPStatus.SERVICE_UNAVAILABLE, HTTPStatus.GATEWAY_TIMEOUT, ] for n in range(retries): try: response = requests.get(url) response.raise_for_status() break except HTTPError as exc: code = exc.response.status_code if code in retry_codes: # retry after n seconds time.sleep(n) continue raise  However, in most scenarios, I subclass requests.Session, make a custom HTTPAdapter that handles exponential backoffs, and the above lives in an overridden requests.Session.request method. An example of that can be seen here.
Question: I wish to create a plot like the following, where I show some values alongside standard deviations.  I have two sets of values, containing the mean and standard deviation obtained by two different methods. I thought of doing this with seaborn, but I don't know exactly how to do it since the official example uses pandas DataFrame objects, which I'm not familiar with. As an example, consider the following starting code: import seaborn as sns mean_1 = [10, 20, 30, 25, 32, 43] std_1 = [2.2, 2.3, 1.2, 2.2, 1.8, 3.5] mean_2 = [12, 22, 30, 13, 33, 39] std_2 = [2.4, 1.3, 2.2, 1.2, 1.9, 3.5]  Thank you,  G.

Answer: Here is a minimal example to create such a plot with the given data. Thanks to vectorization and broadcasting, working with numpy simplifies the code. import matplotlib.pyplot as plt import numpy as np mean_1 = np.array([10, 20, 30, 25, 32, 43]) std_1 = np.array([2.2, 2.3, 1.2, 2.2, 1.8, 3.5]) mean_2 = np.array([12, 22, 30, 13, 33, 39]) std_2 = np.array([2.4, 1.3, 2.2, 1.2, 1.9, 3.5]) x = np.arange(len(mean_1)) plt.plot(x, mean_1, 'b-', label='mean_1') plt.fill_between(x, mean_1 - std_1, mean_1 + std_1, color='b', alpha=0.2) plt.plot(x, mean_2, 'r-', label='mean_2') plt.fill_between(x, mean_2 - std_2, mean_2 + std_2, color='r', alpha=0.2) plt.legend() plt.show()   Another example: import matplotlib.pyplot as plt import numpy as np import seaborn as sns sns.set() N = 100 x = np.arange(N) mean_1 = 25 + np.random.normal(0.1, 1, N).cumsum() std_1 = 3 + np.random.normal(0, .08, N).cumsum() mean_2 = 15 + np.random.normal(0.2, 1, N).cumsum() std_2 = 4 + np.random.normal(0, .1, N).cumsum() plt.plot(x, mean_1, 'b-', label='mean_1') plt.fill_between(x, mean_1 - std_1, mean_1 + std_1, color='b', alpha=0.2) plt.plot(x, mean_2, 'r--', label='mean_2') plt.fill_between(x, mean_2 - std_2, mean_2 + std_2, color='r', alpha=0.2) plt.legend(title='title') plt.show()   PS: Using matplotlib 3.5 or higher, the line and the fill can be combined in the legend: line_1, = plt.plot(x, mean_1, 'b-') fill_1 = plt.fill_between(x, mean_1 - std_1, mean_1 + std_1, color='b', alpha=0.2) line_2, = plt.plot(x, mean_2, 'r--') fill_2 = plt.fill_between(x, mean_2 - std_2, mean_2 + std_2, color='r', alpha=0.2) plt.margins(x=0) plt.legend([(line_1, fill_1), (line_2, fill_2)], ['Series 1', 'Series 2'], title='title')
Question: I'm trying to improve CI pipeline to prevent situations where SQLAlchemy models are added or changed, but no Alembic migration is written or generated by the commit author from hitting the production branch. alembic --help doesn't seem to provide any helpful commands for this case, yet it already has all the metadata required (target_metadata variable) and the database credentials in env.py to make this happen. What would be the best practice for implementing this check in CI?

Answer: Here's a solution that I use. It's a check that I have implemented as a test. from alembic.autogenerate import compare_metadata from alembic.command import upgrade from alembic.runtime.migration import MigrationContext from alembic.config import Config from models.base import Base def test_migrations_sane(): """ This test ensures that models defined by SQLAlchemy match what alembic migrations think the database should look like. If these are different, then once we have constructed the database via Alembic (via running all migrations) alembic will generate a set of changes to modify the database to match the schema defined by SQLAlchemy models. If these are the same, the set of changes is going to be empty. Which is exactly what we want to check. """ engine = "SQLAlchemy DB Engine instance" try: with engine.connect() as connection: alembic_conf_file = "location of alembic.ini" alembic_config = Config(alembic_conf_file) upgrade(alembic_config, "head") mc = MigrationContext.configure(connection) diff = compare_metadata(mc, Base.metadata) assert diff == [] finally: with engine.connect() as connection: # Resetting the DB connection.execute( """ DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO postgres; GRANT ALL ON SCHEMA public TO public; """ )  EDIT: I noticed you linked a library that's supposed to do the same thing. I gave it a go but it seems like it assumes that the database that it's running the check against has to have had alembic run against it. My solution works against a blank db. EDIT EDIT: Since leaving this answer, I have discovered the pytest-alembic package which offers this functionality and more. I have not used it personally but it seems pretty sweet.
Question: I've scraped some data from web sources and stored it all in a pandas DataFrame. Now, in order harness the powerful db tools afforded by SQLAlchemy, I want to convert said DataFrame into a Table() object and eventually upsert all data into a PostgreSQL table. If this is practical, what is a workable method of going about accomplishing this task?

Answer: Update: You can save yourself some typing by using this method.  If you are using PostgreSQL 9.5 or later you can perform the UPSERT using a temporary table and an INSERT ... ON CONFLICT statement: import sqlalchemy as sa # … with engine.begin() as conn: # step 0.0 - create test environment conn.exec_driver_sql("DROP TABLE IF EXISTS main_table") conn.exec_driver_sql( "CREATE TABLE main_table (id int primary key, txt varchar(50))" ) conn.exec_driver_sql( "INSERT INTO main_table (id, txt) VALUES (1, 'row 1 old text')" ) # step 0.1 - create DataFrame to UPSERT df = pd.DataFrame( [(2, "new row 2 text"), (1, "row 1 new text")], columns=["id", "txt"] ) # step 1 - create temporary table and upload DataFrame conn.exec_driver_sql( "CREATE TEMPORARY TABLE temp_table AS SELECT * FROM main_table WHERE false" ) df.to_sql("temp_table", conn, index=False, if_exists="append") # step 2 - merge temp_table into main_table conn.exec_driver_sql( """\ INSERT INTO main_table (id, txt) SELECT id, txt FROM temp_table ON CONFLICT (id) DO UPDATE SET txt = EXCLUDED.txt """ ) # step 3 - confirm results result = conn.exec_driver_sql("SELECT * FROM main_table ORDER BY id").all() print(result) # [(1, 'row 1 new text'), (2, 'new row 2 text')]
Question: I have a parent dataclass and a sub-dataclass inherits the first class. I've redefined __eq__() method in parent dataclass. But when I compare objects sub-dataclass, it doesn't use the __eq__() method defined in parent dataclass. Why is this happening? How can I fix this? MWE: from dataclasses import dataclass @dataclass class A: name: str field1: str = None def __eq__(self, other): print('A class eq') return self.name == other.name @dataclass class B(A): field2: str = None b1 = B('b', 'b1') b2 = B('b', 'b2') print(b1 == b2) 

Answer: The @dataclass decorator adds a default __eq__ implementation. If you use @dataclass(eq=False)  on class B, it will avoid doing that. See https://docs.python.org/3/library/dataclasses.html
Question: I've started to apply SOLID principles to my projects. All of them are clear for me, except dependency inversion, because in Python we have no change to define variable in type of some class inside another class (or maybe just I don't know). So I've realized Dependency Inversion principle in two forms, and want to know which of them is true, how can I correct them. Here are my codes: d1.py: class IFood: def bake(self, isTendir: bool): pass class Production: def __init__(self): self.food = IFood() def produce(self): self.food.bake(True) class Bread(IFood): def bake(self, isTendir:bool): print("Bread was baked")  d2.py: from abc import ABC, abstractmethod class Food(ABC): @abstractmethod def bake(self, isTendir): pass class Production(): def __init__(self): self.bread = Bread() def produce(self): self.bread.bake(True) class Bread(Food): def bake(self, isTendir:bool): print("Bread was baked") 

Answer: # define a common interface any food should have and implement class IFood: def bake(self): pass def eat(self): pass class Bread(IFood): def bake(self): print("Bread was baked") def eat(self): print("Bread was eaten") class Pastry(IFood): def bake(self): print("Pastry was baked") def eat(self): print("Pastry was eaten") class Production: def __init__(self, food): # food now is any concrete implementation of IFood self.food = food # this is also dependency injection, as it is a parameter not hardcoded def produce(self): self.food.bake() # uses only the common interface def consume(self): self.food.eat() # uses only the common interface  Use it: ProduceBread = Production(Bread()) ProducePastry = Production(Pastry())
Question: How may I introduce a continuous hue to my seaborn pairplots?  I am passing in a pandas data frame train_df in order to visualise the relationship between the multiple features.  However I'd also like to add a hue which would use their corresponding target values, target_df. These target values are on a continuous scale (~ floats between 10 and 100).  I have defined a sns.color_palette("RdGr") that I'd like to use.  Right now I have following pairplot (with no hue):  sns.pairplot(train_df)  How can I pass in the target_df as a hue using color palette defined above?  Many thanks in advance. 

Answer: You can just assign the target_df as a column in train_df and pass it as hue: sns.pairplot(data=train_df.assign(target=target_df, hue='target')  However, this will be extremely slow if your target is continuous. Instead, you can do a double for loop: num_features = len(train_df.columns) fig,ax = plt.subplots(num_features, num_features, figsize=(10,10)) for i in train_df.columns: for j in train_df.columns: if i==j: # diagonal sns.distplot(train_df[0], kde=False, ax=ax[i][j]) else: # off diagonal sns.scatterplot(x=train_df[i],y=train_df[j], ax=ax[i][j], hue=target_df, palette='BrBG', legend=False)  Which gives you something like this:
Question: My class has a project where we have to build a database. I keep running into this error where python asks expected a type of the same kind that I gave it (see image ) It says Expected type 'TableEntry', got 'Type[TableEntry]' instead TableEntry is a dataclass instance (as per my assignment). I am only calling it for it's creation newTable = Table(id, TableEntry) Where Table is another dataclass with an id and data with type TableEntry*. *I am required to complete the assignment using this format, and am only asking about the syntax, not how to format it

Answer: When you see an error like this:  Expected type 'TableEntry', got 'Type[TableEntry]' instead  it generally means that in the body of your code you said TableEntry (the name of the type) rather than TableEntry() (an expression that constructs an actual object of that type).
Question: I am preparing a docker image for Ubuntu 20.04 and due to TensorFlow 2.0 requirement, I need Python 3.7. TensorFlow runs on Python 3.5 to 3.7. Running apt install python3 installs Python 3.8 by default and that breaks my TensorFlow installation. Is there any way I can get an apt package for Python 3.7 for Ubuntu 20.04? Since it is going to be inside docker image, I don't want to get into the business of downloading Python 3.7 source code and compiling. Putting those commands in Dockerfile will be overwhelming for me. Is there any simpler way of getting Python 3.7 for Ubuntu 20.04? Running sudo apt-cache madison python3  returns python3 | 3.8.2-0ubuntu2 | http://in.archive.ubuntu.com/ubuntu focal/main amd64 Packages 

Answer: Do you need Ubuntu 20.04? Ubuntu 18.04 comes with Python 3.6, and 3.7 available. If you do, the deadsnakes PPA has Python 3.5-3.7 for Ubuntu 20.04 (Focal). To add it and install: sudo add-apt-repository ppa:deadsnakes/ppa sudo apt-get install python3.7  P.s. I'm not a dev and have no experience with Tensorflow so take this with a grain of salt. (Sidenote: add-apt-repository runs apt-get update automatically, but that's not documented in man add-apt-repository, only add-apt-repository --help. This was fixed in a later release.)
Question: I've been looking for robust type hints for a pandas DataFrame, but cannot seem to find anything useful. This question barely scratches the surface Pythonic type hints with pandas? Normally if I want to hint the type of a function, that has a DataFrame as an input argument I would do: import pandas as pd def func(arg: pd.DataFrame) -> int: return 1  What I cannot seem to find is how do I type hint a DataFrame with mixed dtypes. The DataFrame constructor supports only type definition of the complete DataFrame. So to my knowledge changes in the dtypes can only occur afterwards with the pd.DataFrame().astype(dtypes={}) function. This here works, but doesn't seem very pythonic to me import datetime def func(arg: pd.DataFrame(columns=['integer', 'date']).astype(dtype={'integer': int, 'date': datetime.date})) -> int: return 1  I came across this package: https://pypi.org/project/dataenforce/ with examples such as this one: def process_data(data: Dataset["id": int, "name": object, "latitude": float, "longitude": float]) pass  This looks somewhat promising, but sadly the project is old and buggy. As a data scientist, building a machine learning application with long ETL processes I believe that type hints are important. What do you use and does anybody type hint their dataframes in pandas?

Answer: I have now found the pandera library that seems very promising: https://github.com/pandera-dev/pandera It allows users to create schemas and use those schemas to create verbose checks. From their docs: https://pandera.readthedocs.io/en/stable/schema_models.html import pandas as pd import pandera as pa from pandera.typing import Index, DataFrame, Series class InputSchema(pa.SchemaModel): year: Series[int] = pa.Field(gt=2000, coerce=True) month: Series[int] = pa.Field(ge=1, le=12, coerce=True) day: Series[int] = pa.Field(ge=0, le=365, coerce=True) class OutputSchema(InputSchema): revenue: Series[float] @pa.check_types def transform(df: DataFrame[InputSchema]) -> DataFrame[OutputSchema]: return df.assign(revenue=100.0) df = pd.DataFrame({ "year": ["2001", "2002", "2003"], "month": ["3", "6", "12"], "day": ["200", "156", "365"], }) transform(df) invalid_df = pd.DataFrame({ "year": ["2001", "2002", "1999"], "month": ["3", "6", "12"], "day": ["200", "156", "365"], }) transform(invalid_df)  Also a note from them:  Due to current limitations in the pandas library (see discussion here), pandera annotations are only used for run-time validation and cannot be leveraged by static-type checkers like mypy. See the discussion here for more details.  But still, even though there is no static-type checking I think that this is going in a very good direction.
Question: So the final Python 2 release is out. However, I can't find anywhere what has changed with this release. The corresponding news page on GitHub is also empty. Can anyone shed some light on this?

Answer: Nice question. You can find out for yourself by downloading and comparing the source code both for 2.7.17 and 2.7.18. Since 2.7 is my favorite flavor of Python I've decided to do it myself; here's a WinMerge screenshot:  Looks like there are some differences. On the other hand, Misc\NEWS clearly states:  What's New in Python 2.7.18 final? Release date: 2020-04-19 There were no new changes in version 2.7.18.  That, IMHO, is not completely true but before getting into it let's first see what most of those differences are about. Again, the changes are listed inside Misc\NEWS and they relate to version 2.7.18rc1:  What's New in Python 2.7.18 release candidate 1? Release date: 2020-04-04 Security  bpo-38945: Newline characters have been escaped when performing uu encoding to prevent them from overflowing into to content section of the encoded file. This prevents malicious or accidental modification of data during the decoding process.  bpo-38804: Fixes a ReDoS vulnerability in :mod:http.cookiejar. Patch by Ben Caller.   Core and Builtins  bpo-38535: Fixed line numbers and column offsets for AST nodes for calls without arguments in decorators.  Library  bpo-38576: Disallow control characters in hostnames in http.client, addressing CVE-2019-18348. Such potentially malicious header injection URLs now cause a InvalidURL to be raised.  bpo-27973: Fix urllib.urlretrieve failing on subsequent ftp transfers from the same host.   Build  bpo-38730: Fix problems identified by GCC's -Wstringop-truncation warning.  Windows  bpo-37025: AddRefActCtx() was needlessly being checked for failure in PC/dl_nt.c.  macOS  bpo-38295: Prevent failure of test_relative_path in test_py_compile on macOS Catalina.  C API  bpo-38540: Fixed possible leak in :c:func:PyArg_Parse and similar functions for format units "es#" and "et#" when the macro :c:macro:PY_SSIZE_T_CLEAN is not defined.   So, it looks like 2.7.18 is just a wrap-around of 2.7.18rc1 but is that all? To be exact, there are a few changes that belong to 2.7.18 and have to do with marking it as the final release of Python 2 and warn the user about the EOL of the product. For example, in tools\templates\layout.html you can find the block below:  {% block header %} {%- if outdated %} {% trans %}This document is for an old version of Python that is {% endtrans %}{% trans %}no longer supported{% endtrans %}. {% trans %}You should upgrade and read the {% endtrans %} {% trans %} Python documentation for the current stable release{% endtrans %}. {%- endif %} {% endblock %}  So, I believe, the reasoning behind that final version is twofold: (a) finalize the changes of 2.7.18rc1 and (b) mark it as the final release of Python 2 and let the user know about it.
Question: I have this code: import torch list_of_tensors = [ torch.randn(3), torch.randn(3), torch.randn(3)] tensor_of_tensors = torch.tensor(list_of_tensors)  I am getting the error:  ValueError: only one element tensors can be converted to Python scalars  How can I convert the list of tensors to a tensor of tensors in pytorch?

Answer: Here is a solution: tensor_of_tensors = torch.stack((list_of_tensors)) print(tensor_of_tensors) #shape (3,3)