eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to change the CRS of a raster tif file. When I assign new CRS using the following code: with rio.open(solar_path, mode='r+') as raster: raster.crs = rio.crs.CRS({'init': 'epsg:27700'}) show((raster, 1)) print(raster.crs)  The print function returns 'EPSG:27700', however after plotting the image the CRS clearly hasn't change?

Answer: Unlike Geopandas, rasterio requires manual re-projection when changing the crs. def reproject_raster(in_path, out_path): """ """ # reproject raster to project crs with rio.open(in_path) as src: src_crs = src.crs transform, width, height = calculate_default_transform(src_crs, crs, src.width, src.height, *src.bounds) kwargs = src.meta.copy() kwargs.update({ 'crs': crs, 'transform': transform, 'width': width, 'height': height}) with rio.open(out_path, 'w', **kwargs) as dst: for i in range(1, src.count + 1): reproject( source=rio.band(src, i), destination=rio.band(dst, i), src_transform=src.transform, src_crs=src.crs, dst_transform=transform, dst_crs=crs, resampling=Resampling.nearest) return(out_path)
Question: I'm getting deprecation warning from my pipelines at circleci. Message. /home/circleci/evobench/env/lib/python3.7/site-packages/_pytest/junitxml.py:436: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0.  Command - run: name: Tests command: | . env/bin/activate mkdir test-reports python -m pytest --junitxml=test-reports/junit.xml  How should I modify command to use xunit? Is it possible to a default tool, as it is mentioned in the message? I mean without specyfing xunit or junit. Here's full pipeline.

Answer: Run your command in this ways. with xunit2 python -m pytest -o junit_family=xunit2 --junitxml=test-reports/junit.xml with xunit1 python -m pytest -o junit_family=xunit1 --junitxml=test-reports/junit.xml or python -m pytest -o junit_family=legacy --junitxml=test-reports/junit.xml  This here describes the change in detail:  The default value of junit_family option will change to xunit2 in pytest 6.0, given that this is the version supported by default in modern tools that manipulate this type of file. In order to smooth the transition, pytest will issue a warning in case the --junitxml option is given in the command line but junit_family is not explicitly configured in pytest.ini: PytestDeprecationWarning: The `junit_family` default value will change to 'xunit2' in pytest 6.0. Add `junit_family=legacy` to your  pytest.ini file to silence this warning and make your suite compatible. In order to silence this warning, users just need to configure the junit_family option explicitly: [pytest] junit_family=legacy
Question:  Is there any way that I can change the shap plot background color or text color in the dark theme? I need either the white background or white text. The plot is an object of IPython.core.display.HTML. It is generated by shap.force_plot(explainer.expected_value[1], shap_values[1][0,:], X_test.iloc[0,:],link="logit")  Thanks for the help!

Answer: After adding the argument of matplotlib=True, the problem is solved. shap.force_plot(explainer.expected_value[1], shap_values[1][0,:], X_test.iloc[0,:],link="logit", matplotlib=True)  It seems the plot is created with matplotlib on a html file. To plot properly, firstly I used style.use('seaborn-dark') in the dark model of Jupyter notebook.
Question: I created an AWS Lambda function using Python 3.8 with a Lambda Proxy API Gateway trigger:  It is indeed possible to return custom HTTP error codes: def lambda_handler(event, context): return { 'statusCode': 400, 'body': json.dumps('This is a bad request!') }  However, some online examples (e.g. 1, 2) simply raise an Exception to return the error with a custom message. If an Exception is thrown on the handler as below, the server return a 502 as the response is not in the expected format for Proxy Integration: def lambda_handler(event, context): raise Exception('This is an exception!')  I think the examples rely on some Integration Response templates and do not use the Proxy Integration. Is it possible to achieve the same thing with Lambda Proxy? I'd like to avoid to globally catch exceptions in the Lambda handler to build custom error responses.

Answer: DurandA - I believe you are absolutely correct: the simplified Lambda Proxy Integration approach relies on you catching your exceptions and returning the standardized format: def lambda_handler(event, context): return { 'statusCode': 400, 'body': json.dumps('This is a bad request!') }  The simplified Lambda Proxy Integration feature was announced in a September 2016 blog post, but one of the examples you cited was posted earlier, in a June 2016 blog post, back when the more complicated Integration Response method was the only way. Maybe you have stumbled on an out of date example. You also posted a link to the product documentation for error handling, at the top in the section covering the Lambda proxy integration feature, it says:  With the Lambda proxy integration, Lambda is required to return an output of the following format: { "isBase64Encoded" : "boolean", "statusCode": "number", "headers": { ... }, "body": "JSON string" }   Here is a working example that returns a HTTP 400 with message "This is an exception!" using Lambda Proxy Integration. import json def exception_handler(e): # exception to status code mapping goes here... status_code = 400 return { 'statusCode': status_code, 'body': json.dumps(str(e)) } def lambda_handler(event, context): try: raise Exception('This is an exception!') return { 'statusCode': 200, 'body': json.dumps('This is a good request!') } except Exception as e: return exception_handler(e)  Output from the above: $ http https://**********.execute-api.us-east-2.amazonaws.com/test HTTP/1.1 400 Bad Request Connection: keep-alive Content-Length: 23 Content-Type: application/json Date: Sun, 23 Feb 2020 05:06:59 GMT X-Amzn-Trace-Id: Root=1-********-************************;Sampled=0 x-amz-apigw-id: **************** x-amzn-RequestId: ********-****-****-****-************ "This is an exception!"  I understand your frustration that you do not want to build a custom exception handler. Fortunately, you only have to build a single handler wrapping your lambda_handler function. Wishing you all the best!
Question: Our team uses software that is heavily reliant on dumping NumPy data into files, which slows our code quite a lot. If we could store our NumPy arrays directly in PostgreSQL we would get a major performance boost. Other performant methods of storing NumPy arrays in any database or searchable database-like structure are welcome, but PostgresSQL would be preferred.  My question is very similar to one asked previously. However, I am looking for a more robust and performant answer and I wish to store any arbitrary NumPy array. 

Answer: Not sure if this is what you are after, but assuming you have read/write access to an existing postgres DB: import numpy as np import psycopg2 as psy import pickle db_connect_kwargs = { 'dbname': '<YOUR_DBNAME>', 'user': '<YOUR_USRNAME>', 'password': '<YOUR_PWD>', 'host': '<HOST>', 'port': '<PORT>' } connection = psy.connect(**db_connect_kwargs) connection.set_session(autocommit=True) cursor = connection.cursor() cursor.execute( """ DROP TABLE IF EXISTS numpy_arrays; CREATE TABLE numpy_arrays ( uuid VARCHAR PRIMARY KEY, np_array_bytes BYTEA ) """ )  The gist of this approach is to store any numpy array (of arbitrary shape and data type) as a row in the numpy_arrays table, where uuid is a unique identifier to be able to later retrieve the array. The actual array would be saved in the np_array_bytes column as bytes.  Inserting into the database: some_array = np.random.rand(1500,550) some_array_uuid = 'some_array' cursor.execute( """ INSERT INTO numpy_arrays(uuid, np_array_bytes) VALUES (%s, %s) """, (some_array_uuid, pickle.dumps(some_array)) )  Querying from the database: uuid = 'some_array' cursor.execute( """ SELECT np_array_bytes FROM numpy_arrays WHERE uuid=%s """, (uuid,) ) some_array = pickle.loads(cursor.fetchone()[0])  Performance?  If we could store our NumPy arrays directly in PostgreSQL we would get a major performance boost.  I haven't benchmarked this approach in any way, so I can't confirm nor refute this... Disk Space? My guess is that this approach takes as much disk space as dumping the arrays to a file using np.save('some_array.npy', some_array). If this is an issue consider compressing the bytes before insertion.
Question: I currently have a flask web application using uWSGI web server that implements the WSGI standard and need to migrate this app to uvicorn web server that implements the ASGI standard. If I choose to use uvicorn web server from the many available options say, Hypercorn, Daphne, then which web microframework(instead of flask) should I opt for from the available options say, Starlette, Quart, Django/Channels to get this migration done smoothly? The hierarchy is like:   Uvicorn: an ASGI server Starlette: (uses Uvicorn) a web microframework FastAPI: (uses Starlette) an API microframework with several additional features for building APIs, with data validation, etc.   As what I have read so far,  Quart is a Python web microframework based on Asyncio. It is intended to provide the easiest way to use asyncio in a web context, especially with existing Flask apps.  and  FastAPI has shown to be a Python web framework with one of the best performances, as measured by third-party benchmarks, thanks to being based on and powered by Starlette. https://fastapi.tiangolo.com/benchmarks/  Please suggest with the best approach

Answer: So here I would like to add something that I have concluded so far,  FastAPI learned from Flask (and several of its plug-ins) several things, including its simplicity. For example, the way you declare routes is very similar. That makes it easy to migrate from Flask to FastAPI (which I see a lot of people doing).  Flask is a framework based on the current/old standard for Python web frameworks: WSGI. FastAPI is based on Starlette, which uses the newer standard for asynchronous web frameworks: ASGI. Starlette would be more comparable to Flask, in being a pure “micro-framework”. Almost everything that you can do with Flask, you can do with Starlette (and so with FastAPI). Nevertheless, Starlette has some features not available in Flask (nor in many other WSGI frameworks, like Django, at least by default), like WebSockets, background tasks and others. As FastAPI is based on Starlette, it inherits all its features. Including WebSockets, GraphQL support, templates, etc. So, at the minimum, with FastAPI you can do virtually everything you can with Flask. FastAPI is also a micro-framework (maybe mini-framework, as it includes some extra features for APIs). So, you can structure the project however you want, in many cases, you can even use most of the same files from a Flask project (I’ve done that). This has been explained in this amazing post: https://www.quora.com/What-are-the-advantages-of-using-FastAPI-over-flask Also, Docker image with Uvicorn managed by Gunicorn for high-performance FastAPI web applications in Python 3.7 and 3.6 with performance auto-tuning can be used for minimal implementation. https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker
Question: I have a class Table: @dataclass(frozen=True, eq=True) class Table: name: str signature: Dict[str, Type[DBType]] prinmary_key: str foreign_keys: Dict[str, Type[ForeignKey]] indexed: List[str]  And need to create such dictionary:  table = Table(*args) {table: 'id'}   TypeError: unhashable type: 'dict'  Don't understand what's the problem.

Answer: The autogenerated hash method isn't safe, since it tries to hash the unhashable attributes signature, primary_key, and indexed. You need to define your own __hash__ method that ignores those attributes. One possibility is def __hash__(self): return hash((self.name, self.primary_key))  Both self.name and self.primary_key are immutable, so a tuple containing those values is also immutable and thus hashable.  An alternative to defining this method explicitly would be to use the field function to turn off the mutable fields for hashing purposes. @dataclass(frozen=True, eq=True) class Table: name: str signature: Dict[str, Type[DBType]] = field(compare=False) prinmary_key: str foreign_keys: Dict[str, Type[ForeignKey]] = field(compare=False) indexed: List[str] = field(compare=False)  field has a hash parameter whose default value is the value of compare, and the documentation discourages using a different value for hash. (Probably to ensure that equal items hash identically.) It's unlikely that you really want to use these three fields for the purposes of comparing two tables, so you this should be OK. I would consult the documentation rather than relying on my relatively uninformed summary of it.
Question: When using os.getenv to retrieve environment variables, the default behavior returns a type of Optional[str]. This is problematic as any downstream methods/functions that utilize these variables will likely be defined to accept a str type explicitly. Is there an accepted usage to get around this issue or enforce the str return type? In the stub file definition for getenv in typeshed you can find that getenv can have a return type of Optional[str] or Union[str, T_] depending on the usage of the default kwarg.  The four options I can see as yet are:  Define any downstream operations to accept Optional[str] types as arguments. This doesn't feel particularly right as a function/method may not be structured in a way that the Optional type makes sense. i.e. the operation has no reason for a particular argument to be None. Use the default kwarg for getenv and provide a str default value. This seems more correct, but requires that one set a default value for every usage of getenv. The only problem I can see with this is that doing so may be confounding for testing or usage in different environments. Define some kind of variable checking function. This could be a function that accepts the name of an environment variable to load, explicitly returns a string, and raises an error if the environment variable doesn't exist. Explicitly set the type of the value returned by getenv to be a str. I really don't like this as it expects the environment to always be properly configured which, in my experience, is not a good assumption.  Find below an example that raises a mypy error. import os SOME_VAR = os.getenv("SOME_VAR") def some_func(val: str) -> None: print(f"Loaded env var: {val}") some_func(SOME_VAR)  The above raises the mypy error:  error: Argument 1 to "some_func" has incompatible type "Optional[str]"; expected "str" 

Answer: tl;dr Use os.environ['SOME_VAR'] if you're sure it's always there  os.getenv can and does return None -- mypy is being helpful in showing you have a bug there: >>> repr(os.getenv('DOES_NOT_EXIST')) 'None' >>> repr(os.getenv('USER')) "'asottile'"  Alternatively, you can convince mypy that it is of the type you expect in two different ways:  utilizing assertions:  x = os.getenv('SOME_VAR') assert x is not None, x # mypy will believe that it is non-None after this point   utilizing a cast: from typing import cast x = cast(str, os.getenv('SOME_VAR')) # mypy will believe that it is a `str` after this point   (the cast has some downsides in that it is never checked, whereas the assertion will hopefully lead to a test failure) I would suggest not ignoring this error / working around it and instead use os.environ['SOME_VAR'] for things you expect to always be there, or write a condition to check for the error case when it is missing
Question: I am writing a Python program to animate a tangent line along a 3D curve. However, my tangent line is not moving. I think the problem is  line.set_data(np.array(Tangent[:,0]).T,np.array(Tangent[:,1]).T) in animate(i) but I can't figure out. Any help will be appreciated. The following is the code. from mpl_toolkits import mplot3d import numpy as np import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation import matplotlib matplotlib.use( 'tkagg' ) plt.style.use('seaborn-pastel') fig = plt.figure() ax = plt.axes(projection='3d') ax = plt.axes(projection='3d') # Data for a three-dimensional line zline = np.linspace(0, 15, 1000) xline = np.sin(zline) yline = np.cos(zline) ax.plot3D(xline, yline, zline, 'red') def curve(t): return [np.sin(t),np.cos(t),t] def vector_T(t): T = [np.cos(t),-np.sin(t),1] return T/np.linalg.norm(T) len = 2 def tangent_line(t): P = np.add(curve(t),len*vector_T(t)) Q = np.subtract(curve(t),len*vector_T(t)) return np.array([P, Q]).T t0 = 0 Tangent=tangent_line(t0) line, = ax.plot3D(Tangent[0], Tangent[1], Tangent[2], 'green') def init(): line.set_data([], []) return line, def animate(i): t0 = 15* (i/200) Tangent=tangent_line(t0) #print(Tangent) line.set_data(np.array(Tangent[:,0]).T,np.array(Tangent[:,1]).T) return line, anim = FuncAnimation(fig, animate, init_func=init, frames=200, interval=20, blit=True) plt.show() 

Answer: you've called the wrong function in animate: Replace line.set_data(...) with line.set_data_3d(Tangent[0], Tangent[1], Tangent[2]) and it will work. There are still some minor issues in the code (e.g., don't use len as a variable name). I'd recommend using the following: #!/usr/bin/env python3 from mpl_toolkits import mplot3d import numpy as np import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation import matplotlib matplotlib.use('tkagg') plt.style.use('seaborn-pastel') fig = plt.figure() ax = plt.axes(projection='3d') # Data for a three-dimensional line zline = np.linspace(0, 15, 1000) xline = np.sin(zline) yline = np.cos(zline) ax.plot3D(xline, yline, zline, 'red') def curve(t): return [ np.sin(t), np.cos(t), t ] def tangent(t): t = [ np.cos(t), -np.sin(t), 1.0 ] return t/np.linalg.norm(t) def tangent_line(t): length = 2.0 offset = length * tangent(t) pos = curve(t) return np.array([ pos-offset, pos+offset ]).T line = ax.plot3D(*tangent_line(0), 'green')[0] def animate(i): line.set_data_3d(*tangent_line(15* (i/200))) return [ line ] anim = FuncAnimation(fig, animate, frames=200, interval=20, blit=True) plt.show()
Question: With Pandas 1.0.1, I'm unable to merge if the  df = df.merge(df2, on=some_column)  yields File /home/torstein/code/fintechdb/Sheets/sheets/gild.py, line 42, in gild df = df.merge(df2, on=some_column) File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py, line 7297, in merge validate=validate, File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 88, in merge return op.get_result() File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 643, in get_result join_index, left_indexer, right_indexer = self._get_join_info() File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 862, in _get_join_info (left_indexer, right_indexer) = self._get_join_indexers() File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 841, in _get_join_indexers self.left_join_keys, self.right_join_keys, sort=self.sort, how=self.how File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 1311, in _get_join_indexers zipped = zip(*mapped) File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 1309, in <genexpr> for n in range(len(left_keys)) File /home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py, line 1918, in _factorize_keys rlab = rizer.factorize(rk) File pandas/_libs/hashtable.pyx, line 77, in pandas._libs.hashtable.Factorizer.factorize File pandas/_libs/hashtable_class_helper.pxi, line 1817, in pandas._libs.hashtable.PyObjectHashTable.get_labels File pandas/_libs/hashtable_class_helper.pxi, line 1732, in pandas._libs.hashtable.PyObjectHashTable._unique File pandas/_libs/missing.pyx, line 360, in pandas._libs.missing.NAType.__bool__ TypeError: boolean value of NA is ambiguous  while this works: df[some_column].fillna(np.nan, inplace=True) df2[some_column].fillna(np.nan, inplace=True) df = df.merge(df2, on=some_column) # Works  If instead, I do df[some_column].fillna(pd.NA, inplace=True)  then the error returns.

Answer: This has to do with pd.NA being implemented in pandas 1.0.0 and how the pandas team decided it should work in a boolean context. Also, you take into account it is an experimental feature, hence it shouldn't be used for anything but experimenting:  Warning Experimental: the behaviour of pd.NA can still change without warning.  In another link of pandas documentation, where it covers working with missing values, is where I believe the reason and the answer you are looking for can be found:  NA in a boolean context: Since the actual value of an NA is unknown, it is ambiguous to convert NA to a boolean value. The following raises an error: TypeError: boolean value of NA is ambiguous  Furthermore, it provides a valuable piece of advise: "This also means that pd.NA cannot be used in a context where it is evaluated to a boolean, such as if condition: ... where condition can potentially be pd.NA. In such cases, isna() can be used to check for pd.NA or condition being pd.NA can be avoided, for example by filling missing values beforehand."
Question: I have a simple function that connects to a DB and fetches some data. db.py from sqlalchemy import create_engine from sqlalchemy.pool import NullPool def _create_engine(app): impac_engine = create_engine( app['DB'], poolclass=NullPool # this setting enables NOT to use Pooling, preventing from timeout issues. ) return impac_engine def get_all_pos(app): engine = _create_engine(app) qry = """SELECT DISTINCT id, name FROM p_t ORDER BY name ASC""" try: cursor = engine.execute(qry) rows = cursor.fetchall() return rows except Exception as re: raise re  I'm trying to write some test cases by mocking this connection -  tests.py import unittest from db import get_all_pos from unittest.mock import patch from unittest.mock import Mock class TestPosition(unittest.TestCase): @patch('db.sqlalchemy') def test_get_all_pos(self, mock_sqlalchemy): mock_sqlalchemy.create_engine = Mock() get_all_pos({'DB': 'test'}) if __name__ == '__main__': unittest.main()  When I run the above file python tests.py, I get the following error -   "Could not parse rfc1738 URL from string '%s'" % name sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string 'test'  Shouldn't mock_sqlalchemy.create_engine = Mock() give me a mock object and bypass the URL check. 

Answer: Another option would be to mock your _create_engine function. Since this is a unit test and we want to test get_all_pos we shouldn't need to rely on the behavior of _create_engine, so we can just patch that like so. import unittest import db from unittest.mock import patch class TestPosition(unittest.TestCase): @patch.object(db, '_create_engine') def test_get_all_pos(self, mock_sqlalchemy): args = {'DB': 'test'} db.get_all_pos(args) mock_sqlalchemy.assert_called_once() mock_sqlalchemy.assert_called_with({'DB': 'test'}) if __name__ == '__main__': unittest.main()  If you want to test certain results you will need to properly set all the corresponding attributes. I would recommend not chaining it into one call so that it is more readable as shown below. import unittest import db from unittest.mock import patch from unittest.mock import Mock class Cursor: def __init__(self, vals): self.vals = vals def fetchall(self): return self.vals class TestPosition(unittest.TestCase): @patch.object(db, '_create_engine') def test_get_all_pos(self, mock_sqlalchemy): to_test = [1, 2, 3] mock_cursor = Mock() cursor_attrs = {'fetchall.return_value': to_test} mock_cursor.configure_mock(**cursor_attrs) mock_execute = Mock() engine_attrs = {'execute.return_value': mock_cursor} mock_execute.configure_mock(**engine_attrs) mock_sqlalchemy.return_value = mock_execute args = {'DB': 'test'} rows = db.get_all_pos(args) mock_sqlalchemy.assert_called_once() mock_sqlalchemy.assert_called_with({'DB': 'test'}) self.assertEqual(to_test, rows)