eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a model class Entry(models.Model): maca = models.CharField(max_length=100, blank=True, null=True)  This field will accept only numbers (cannot set char field to integer field for business reasons) Now I have to get all entries that have maca greater than 90 What I'm trying to do is this: Entry.objects.filter(maca__gte=90)  But get isn't working because the maca is a string. How can I convert maca to int before filtering? Or something like this? Thanks

Answer: Try this: from django.db.models.functions import Cast from django.db.models import IntegerField Entry.objects.annotate(maca_integer=Cast('maca', output_field=IntegerField())).filter(maca_integer__gte=90)
Question: I'm essentially making a pinger, that makes has a 2d list, of key / webhook pairs, and after pinging a key, send the response to a webhook the 2d list goes as follows: some_list = [["key1", "webhook1"], ["key2", "webhook2"]]  My program is essentially a loop, and I'm not too sure how I can rotate the some_list data, in the function. Here's a little demo of what my script looks like: async def do_ping(some_pair): async with aiohttp.ClientSession() as s: tasks = await gen_tasks(s, some_pair) results = await asyncio.gather(*tasks*) sleep(10) await do_ping(some_pair)  I've tried: async def main(): for entry in some_list: asyncio.run(do_ping(entry))  but due to the do_ping function being a self-calling loop, it just calls the first one over and over again, and never gets to the ones after it. Hoping to find a solution to this, whether it's threading or alike, and if you have a better way of structuring some_list values (which I assume would be a dictionary), feel free to drop that feedback as well

Answer: You made your method recursive await do_ping(some_pair), it never ends for the loop in main to continue. I would restructure the application like this: async def do_ping(some_pair): async with aiohttp.ClientSession() as s: while True: tasks = await gen_tasks(s, some_pair) results = await asyncio.gather(*tasks) await asyncio.sleep(10) async def main(): tasks = [do_ping(entry) for entry in some_list] await asyncio.gather(*tasks) if __name__ == "__main__": asyncio.run(main())  Alternatively you could move the repeat and sleeping logic into the main: async def do_ping(some_pair): async with aiohttp.ClientSession() as s: tasks = await gen_tasks(s, some_pair) results = await asyncio.gather(*tasks) async def main(): while True: tasks = [do_ping(entry) for entry in some_list] await asyncio.gather(*tasks) await asyncio.sleep(10) if __name__ == "__main__": asyncio.run(main())  You could also start the tasks before doing a call to sleep, and gather them afterwards. That would make the pings more consistently start at 10 second intervals instead of being 10 seconds + the time it takes to gather the results: async def main(): while True: tasks = [ asyncio.create_task(do_ping(entry)) for entry in some_list ] await asyncio.sleep(10) await asyncio.wait(tasks)   EDIT As pointed out by creolo you should only create a single ClientSession object. See https://docs.aiohttp.org/en/stable/client_reference.html  Session encapsulates a connection pool (connector instance) and supports keepalives by default. Unless you are connecting to a large, unknown number of different servers over the lifetime of your application, it is suggested you use a single session for the lifetime of your application to benefit from connection pooling.  async def do_ping(session, some_pair): tasks = await gen_tasks(session, some_pair) results = await asyncio.gather(*tasks) async def main(): async with aiohttp.ClientSession() as session: while True: tasks = [ asyncio.create_task(do_ping(session, entry)) for entry in some_list ] await asyncio.sleep(10) await asyncio.wait(tasks)
Question: Is there a possibility in polars to read in csv with german number formatting like it is possible in pandas.read_csv() with the parameters "decimal" and "thousands"

Answer: Currently, the Polars read_csv method does not expose those parameters. However, there is an easy workaround to convert them. For example, with this csv, allow Polars to read the German-formatted numbers as utf8. import polars as pl my_csv = b"""col1\tcol2\tcol3 1.234,5\tabc\t1.234.567 9.876\tdef\t3,21 """ df = pl.read_csv(my_csv, separator="\t") print(df)  shape: (2, 3) ┌─────────┬──────┬───────────┐ │ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞═════════╪══════╪═══════════╡ │ 1.234,5 ┆ abc ┆ 1.234.567 │ │ 9.876 ┆ def ┆ 3,21 │ └─────────┴──────┴───────────┘  From here, the conversion is just a few lines of code: df = df.with_columns( pl.col("col1", "col3") .str.replace_all(r"\.", "") .str.replace(",", ".") .cast(pl.Float64) # or whatever datatype needed ) print(df)  shape: (2, 3) ┌────────┬──────┬────────────┐ │ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- │ │ f64 ┆ str ┆ f64 │ ╞════════╪══════╪════════════╡ │ 1234.5 ┆ abc ┆ 1.234567e6 │ │ 9876.0 ┆ def ┆ 3.21 │ └────────┴──────┴────────────┘  Just be careful to apply this logic only to numbers encoded in German locale. It will mangle numbers formatted in other locales.
Question: Consider a Polars data frame with a column of str type that indicates the date in the format '27 July 2020'. I would like to convert this column to the polars.datetime type, which is distinct from the Python standard datetime. import polars as pl from datetime import datetime df = pl.DataFrame({ "id": [1, 2], "event_date": ["27 July 2020", "31 December 2020"] }) df = df.with_columns( pl.col("event_date").map_elements(lambda x: x.replace(" ", "-")) .map_elements(lambda x: datetime.strptime(x, "%d-%B-%Y")) )  shape: (2, 2) ┌─────┬─────────────────────┐ │ id ┆ event_date │ │ --- ┆ --- │ │ i64 ┆ datetime[μs] │ ╞═════╪═════════════════════╡ │ 1 ┆ 2020-07-27 00:00:00 │ │ 2 ┆ 2020-12-31 00:00:00 │ └─────┴─────────────────────┘  Suppose we try to process df further to create a new column indicating the quarter of the year an event took place. df.with_columns( pl.col("event_date").map_elements(lambda x: x.month) .map_elements(lambda x: 1 if x in range(1,4) else 2 if x in range(4,7) else 3 if x in range(7,10) else 4) .alias("quarter") )  shape: (2, 3) ┌─────┬─────────────────────┬─────────┐ │ id ┆ event_date ┆ quarter │ │ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs] ┆ i64 │ ╞═════╪═════════════════════╪═════════╡ │ 1 ┆ 2020-07-27 00:00:00 ┆ 3 │ │ 2 ┆ 2020-12-31 00:00:00 ┆ 4 │ └─────┴─────────────────────┴─────────┘  How would I do this in Polars without applying custom lambdas through map_elements?

Answer: The easiest way to convert strings to Date/Datetime is to use Polars' own functions:  .str.to_date() .str.to_datetime()  df.with_columns( pl.col("event_date").str.to_datetime("%d %B %Y") )  shape: (2, 2) ┌─────┬─────────────────────┐ │ id ┆ event_date │ │ --- ┆ --- │ │ i64 ┆ datetime[μs] │ ╞═════╪═════════════════════╡ │ 1 ┆ 2020-07-27 00:00:00 │ │ 2 ┆ 2020-12-31 00:00:00 │ └─────┴─────────────────────┘  The Temporal section of the docs shows the supported functions in the .dt namespace. In the case of your second example, there is a dedicated quarter expression:  .dt.quarter()  df = df.with_columns( pl.col("event_date").str.to_datetime("%d %B %Y") ).with_columns( pl.col("event_date").dt.quarter().alias("quarter") )  shape: (2, 3) ┌─────┬─────────────────────┬─────────┐ │ id ┆ event_date ┆ quarter │ │ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs] ┆ i8 │ ╞═════╪═════════════════════╪═════════╡ │ 1 ┆ 2020-07-27 00:00:00 ┆ 3 │ │ 2 ┆ 2020-12-31 00:00:00 ┆ 4 │ └─────┴─────────────────────┴─────────┘
Question: I want to try out polars in Python so what I want to do is concatenate several dataframes that are read from jsons. When I change the index to date and have a look at lala1.head() I see that the column date is gone, so I basically lose the index. Is there a better solution or do I need to sort by date, which basically does the same as setting the index to date? import polars as pl quarterly_balance_df = pl.read_json('../AAPL/single_statements/1985-09-30-quarterly_balance.json') q1 = quarterly_balance_df.lazy().with_columns(pl.col("date").str.to_date()) quarterly_balance_df = q1.collect() q2 = quarterly_balance_df.lazy().with_columns(pl.col("fillingDate").str.to_date()) quarterly_balance_df = q2.collect() q3 = quarterly_balance_df.lazy().with_columns(pl.col("acceptedDate").str.to_date()) quarterly_balance_df = q3.collect() quarterly_balance_df2 = pl.read_json('../AAPL/single_statements/1986-09-30-quarterly_balance.json') q1 = quarterly_balance_df2.lazy().with_columns(pl.col("date").str.to_date()) quarterly_balance_df2 = q1.collect() q2 = quarterly_balance_df2.lazy().with_columns(pl.col("fillingDate").str.to_date()) quarterly_balance_df2 = q2.collect() q3 = quarterly_balance_df2.lazy().with_columns(pl.col("acceptedDate").str.to_date()) quarterly_balance_df2 = q3.collect() lala1 = pl.from_pandas(quarterly_balance_df.to_pandas().set_index('date')) lala2 = pl.from_pandas(quarterly_balance_df.to_pandas().set_index('date')) test = pl.concat([lala1,lala2]) 

Answer: Polars intentionally eliminates the concept of an index. From the "Coming from Pandas" section in the User Guide:  Polars aims to have predictable results and readable queries, as such we think an index does not help us reach that objective.  Indeed, the from_pandas method ignores any index. For example, if we start with this data: import polars as pl df = pl.DataFrame( { "key": [1, 2], "var1": ["a", "b"], "var2": ["r", "s"], } ) print(df)  shape: (2, 3) ┌─────┬──────┬──────┐ │ key ┆ var1 ┆ var2 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═════╪══════╪══════╡ │ 1 ┆ a ┆ r │ │ 2 ┆ b ┆ s │ └─────┴──────┴──────┘  Now, if we export this Polars dataset to Pandas, set key as the index in Pandas, and then re-import to Polars, you'll see the 'key' column disappear. pl.from_pandas(df.to_pandas().set_index("key"))  shape: (2, 2) ┌──────┬──────┐ │ var1 ┆ var2 │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪══════╡ │ a ┆ r │ │ b ┆ s │ └──────┴──────┘  This is why your Date column disappeared. In Polars, you can sort, summarize, or join by any set of columns in a DataFrame. No need to declare an index. I recommend looking through the Polars User Guide. It's a great place to start. And there's a section for those coming from Pandas.
Question: I would like to replace Pandas with Polars but I was not able to find out how to use Polars with Plotly without converting to Pandas. I wonder if there is a way to completely cut Pandas out of the process. Consider the following test data: import polars as pl import numpy as np import plotly.express as px df = pl.DataFrame( { "nrs": [1, 2, 3, None, 5], "names": ["foo", "ham", "spam", "egg", None], "random": np.random.rand(5), "groups": ["A", "A", "B", "C", "B"], } ) fig = px.bar(df, x='names', y='random') fig.show()  I would like this code to show the bar chart in a Jupyter notebook but instead it returns an error: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/polars/internals/frame.py:1483: UserWarning: accessing series as Attribute of a DataFrame is deprecated warnings.warn("accessing series as Attribute of a DataFrame is deprecated")  It is possible to transform the Polars data frame to a Pandas data frame with df = df.to_pandas(). Then, it works. However, is there another, simpler and more elegant solution?

Answer: Yes, no need for converting to a Pandas dataframe. Someone (sa-) has requested supporting a better option here and included a workaround for it.  "The workaround that I use right now is px.line(x=df["a"], y=df["b"]), but it gets unwieldy if the name of the data frame is too big"  For the OP's code example, the approach of specifying the dataframe columns explicitly works. I find in addition to specifying the dataframe columns with px.bar(x=df["names"], y=df["random"]) - or - px.bar(df, x=df["names"], y=df["random"]), casting to a list can also work: import polars as pl import numpy as np import plotly.express as px df = pl.DataFrame( { "nrs": [1, 2, 3, None, 5], "names": ["foo", "ham", "spam", "egg", None], "random": np.random.rand(5), "groups": ["A", "A", "B", "C", "B"], } ) px.bar(df, x=list(df["names"]), y=list(df["random"]))  Knowing polars better, you may see some other options once you see the idea of the workaround. The example posted there is simpler, instead of px.line(df, x="a", y="b") like you could use for a Pandas dataframe, you use px.line(x=df["a"], y=df["b"]). With polars, that is: import polars as pl import plotly.express as px df = pl.DataFrame({"a":[1,2,3,4,5], "b":[1,4,9,16,25]}) px.line(x=df["a"], y=df["b"])  (Note that using plotly.express requires Pandas to be installed, see here and here. I used plotly.express in my answer because it was closer to the OP. The code could be adapted to using plotly.graph_objects if there was a desire to not have Pandas installed & involved at all.)
Question: If I have a django queryset print(queryset.query) shows me the SQL statement so I can validate it. But with aggregations they never return a queryset. How do you print those queries out. I guess I can turn on debug logging for the ORM and find them that way but it seems like I should be able to get it right before the execution engine sends it off to postgres.....

Answer: You can use connection.queries after your aggregation query. Example: ... from django.db import connection def your_view(request): # Your view logic and aggregate queryset print(connection.queries) return render(request, 'index.html', {})  The output is a dictionary of SQL queries that you entered into the database.
Question: If I try to validate code that uses pandas methodes with pylance in strict mode, I get a validation error. import pandas as pd ser: pd.Series[float] = pd.Series([.1, .2, .5, .3]) print(ser.max())  Pylance in strict mode returns the error:  Type of "max" is partially unknown  Am I doing something wrong? Can I avoid this error without switching pylance to basic mode?

Answer: According to pyright documentation, one can turn off this specific check by creating pyrightconfig.json file in the same project folder with the next content: { "reportUnknownMemberType":"none", }  So you can be still in strict mode, but with one type of checks disabled.
Question: I'm trying to flatten some mixed arrays in Python using LC. I'm having some trouble figuring out how to structure it. Here's the array's i'm trying to flatten arr_1 = [1, [2, 3], 4, 5] arr_2 = [1,[2,3],[[4,5]]]  I tried this methods for arr_1 but get "TypeError: 'int' object is not iterable" print([item if type(items) is list else items for items in arr_1 for item in items])  So I decided to break it into parts to see where it's failing by using this def check(item): return item; print([check(item) if type(items) is list else check(items) for items in [1, [2, 3], 4, 5] for items in arr_2])  Through the debugger I found that it's failing at the 2d array in for items in [1, [2, 3], 4, 5]  I don't need the LC to be in one line but I just wanted to know how to do it in a single nested LC if its even possible.

Answer: Using an internal stack and iter's second form to simulate a while loop: def flatten(obj): return [x for stack in [[obj]] for x, in iter(lambda: stack and [stack.pop()], []) if isinstance(x, int) or stack.extend(reversed(x))] print(flatten([1, [2, 3], 4, 5])) print(flatten([1, [2, 3], [[4, 5]]])) print(flatten([1, [2, [], 3], [[4, 5]]]))  Output (Try it online!): [1, 2, 3, 4, 5] [1, 2, 3, 4, 5] [1, 2, 3, 4, 5]  Slight variation, splitting the "long" line into two (Try it online!): def flatten(obj): return [x for stack in [[obj]] for _ in iter(lambda: stack, []) for x in [stack.pop()] if isinstance(x, int) or stack.extend(reversed(x))]  To explain it a bit, here's roughly the same with ordinary code: def flatten(obj): result = [] stack = [obj] while stack: x = stack.pop() if isinstance(x, int): result.append(x) else: stack.extend(reversed(x)) return result  If the order doesn't matter, we can use a queue instead (inspired by 0x263A's comment), although it's less memory-efficient (Try it online!): def flatten(obj): return [x for queue in [[obj]] for x in queue if isinstance(x, int) or queue.extend(x)]  We can fix the order if instead of putting each list's contents at the end of the queue, we insert them right after the list (which is less time-efficient) in the "priority" queue (Try it online!): def flatten(obj): return [x for pqueue in [[obj]] for i, x in enumerate(pqueue, 1) if isinstance(x, int) or pqueue.__setitem__(slice(i, i), x)]
Question: I am getting the error  ImportError: cannot import name 'escape' from 'jinja2'  When trying to run code using the following requirements.txt: chart_studio==1.1.0 dash==2.1.0 dash_bootstrap_components==1.0.3 dash_core_components==2.0.0 dash_html_components==2.0.0 dash_renderer==1.9.1 dash_table==5.0.0 Flask==1.1.2 matplotlib==3.4.3 numpy==1.20.3 pandas==1.3.4 plotly==5.5.0 PyYAML==6.0 scikit_learn==1.0.2 scipy==1.7.1 seaborn==0.11.2 statsmodels==0.12.2 urllib3==1.26.7  Tried pip install jinja2  But the requirement is already satisfied. Running this code on a windows system.

Answer: Jinja is a dependency of Flask and Flask V1.X.X uses the escape module from Jinja, however recently support for the escape module was dropped in newer versions of Jinja. To fix this issue, simply update to the newer version of Flask V2.X.X in your requirements.txt where Flask no longer uses the escape module from Jinja. Flask>=2.2.2  Also, do note that Flask V1.X.X is no longer supported by the team. If you want to continue to use this older version, this Github issue may help.
Question: I wanted to install python 3.10 but that version is not available on pyenv version list. checked by pyenv install --list. People suggested to upgrade pyenv that but I do not see help related to updating pyenv.

Answer: pyenv isn't really 'installed' in a traditional sense, it's just a git checkout. All you have to do to update is cd ~/.pyenv git pull  That also updates the list of available python versions.
Question: Mypy infers ORM non-nullable instance attributes as optionals. Filename: test.py from sqlalchemy.orm import decl_api, registry from sqlalchemy import BigInteger, Column, String mapper_registry = registry() class Base(metaclass=decl_api.DeclarativeMeta): __abstract__ = True registry = mapper_registry metadata = mapper_registry.metadata __init__ = mapper_registry.constructor class Person(Base): __tablename__ = "persons" id = Column(BigInteger, primary_key=True, autoincrement=True) name = Column(String(40), nullable=False) def main(person: Person): person_id = person.id person_name = person.name reveal_locals()  Running mypy test.py yields: test.py:27: note: Revealed local types are: test.py:27: note: person: test.Person test.py:27: note: person_id: Union[builtins.int, None] test.py:27: note: person_name: Union[builtins.str, None]  As far as my understanding goes, person_id and person_name should have been int and str respectively since they are set as non-nullable. What am I missing here?  Relevant libraries: SQLAlchemy 1.4.25 sqlalchemy2-stubs 0.0.2a15 mypy 0.910 mypy-extensions 0.4.3 

Answer: I had the same question when evaluating a nullable=False column with mypy. One of my teammates found the answer in the SqlAlchemy docs: https://docs.sqlalchemy.org/en/14/orm/extensions/mypy.html#introspection-of-columns-based-on-typeengine  The types are by default always considered to be Optional, even for the primary key and non-nullable column. The reason is because while the database columns “id” and “name” can’t be NULL, the Python attributes id and name most certainly can be None without an explicit constructor:  There are also mitigations they list in that article with declaring the columns to be mapped types explicitly.
Question: Im using structlog for logging and want the exception/stacktrace to be printed in json format. Currently its not formatted and in string format which is not very readable { "message": "Error info with an exc", "timestamp": "2022-03-31T13:32:33.928188+00:00", "logger": "__main__", "level": "error", "exception": "Traceback (most recent call last):\n File \"../../main.py\", line 21, in <module>\n assert 'foo' == 'bar'\nAssertionError" }  Wanted exception in json format like { "message": "Error info with an exc", "timestamp": "2022-03-31T13:32:33.928188+00:00", "logger": "__main__", "level": "error", "exception": { "File": "../.../main.py", "line": 21, "function": "<module>", "errorStatement": "assert 'foo' == 'bar'", "errorType":"AssertionError", } }  This is just a small example I am aslo using traceback library and passing the stackTrace which gets printed in large string block Do we have any library available which does stacktrace json formatting. or do we have to write a custom function?

Answer: Here is a snippet which does that: https://gitlab.com/-/snippets/2284049 It will eventually land directly in structlog. edit: https://github.com/hynek/structlog/pull/407 has been merged and will be part of v22.1.
Question: I have a new MacBook with the Apple M1 chipset. To install tensorflow, I follow the instructions here, i.e., installing tensorflow-metal and tensorflow-macos instead of the normal tensorflow package. While this works fine, it means that I can't run the typical pip install -r requirements.txt as long as we have tensorflow in the requirements.txt. If we instead include tensorflow-macos, it'll lead to problems for non-M1 or even non-macOS users. Our library must work on all platforms. Is there a generic install command that installs the correct TensorFlow version depending on whether the computer is a M1 Mac or not? So that we can use a single requirements.txt for everyone? Or if that's not possible, can we pass some flag/option, e.g., pip install -r requirements.txt --m1 to install some variation? What's the simplest and most elegant solution here?

Answer: According to this post Is there a way to have a conditional requirements.txt file for my Python application based on platform? You can use conditionals on your requirements.txt, thus tensorflow==2.8.0; sys_platform != 'darwin' or platform_machine != 'arm64' tensorflow-macos==2.8.0; sys_platform == 'darwin' and platform_machine == 'arm64'
Question: When running our lint checks with the Python Black package, an error comes up:  ImportError: cannot import name '_unicodefun' from 'click' (/Users/robot/.cache/pre-commit/repo3u71ccm2/py_env-python3.9/lib/python3.9/site-packages/click/init.py)`  In researching this, I found the following related issues:  ImportError: cannot import name '_unicodefun' from 'click' #2976 ImportError: cannot import name '_unicodefun' from 'click' #6013  How can I solve this problem? Is this a false positive from the linter? Do I need to modify my code?

Answer: This has been fixed by Black 22.3.0. Versions before that won't work with click 8.1.0. Incompatible with click 8.1.0 (ImportError: cannot import name '_unicodefun' from 'click') #2964 E.g.: black.yml  python-version: 3.8 - name: install black run: | - pip install black==20.8b1 + pip install black==22.3.0 - name: run black run: | black . --check --line-length 100  https://github.com/Clinical-Genomics/cgbeacon2/pull/221/files As a workaround, pin click to the last version via pip install --upgrade click==8.0.2.