eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Does it make sense to specify range of allowed Python versions in environment.yml file? I got this idea while reading the Google's Biq Query documentation Supported Python Versions Python >= 3.7, < 3.11  If this makes sense then what is the right syntax to specify the range in the environment.yml file?

Answer: Recommendation: Prefer exact version, not range While there is nothing logically incorrect with specifying a version range for Python, it has the downside of defining a large solution space, which can lead to slow solving. For practical environments, I would recommend specifying the version for python through the minor version, e.g., python=3.9. Note that this behavior mostly pertains to central packages that define variants of other packages, like python, r-base, or cudatoolkit. For most other packages, the impact is not as drastic. Benchmarking Here is a simple environment for basic data analysis, with and without the Python version specified. so-py39.yaml name: so-py39 channels: - conda-forge - nodefaults dependencies: - python ==3.9 - ipykernel - numba - pandas - scikit-learn - scipy  so-py3x.yaml name: so-py3x channels: - conda-forge - nodefaults dependencies: - python >=3.7,<4.0 - ipykernel - numba - pandas - scikit-learn - scipy  Conda First, we can use the regular conda command. Command command time conda env create -dqn foo -f [file]  Results Timing creating environment: so-py39.yaml 20.51 real 19.57 user 0.94 sys 20.73 real 19.84 user 0.97 sys 19.43 real 18.66 user 0.95 sys 19.22 real 18.36 user 0.92 sys 19.34 real 18.48 user 0.94 sys 19.08 real 18.16 user 0.94 sys Timing creating environment: so-py3x.yaml 30.53 real 29.56 user 1.00 sys 29.21 real 28.21 user 1.08 sys 31.13 real 29.77 user 1.07 sys 29.93 real 28.46 user 0.99 sys 30.53 real 29.43 user 0.98 sys 28.60 real 27.68 user 1.03 sys  That is, solving for the environment with a range takes ~10s (~50%) longer. Mamba We can also test solving the environment with Mamba. Command command time mamba env create -dqn foo -f [file]  Results Timing creating environment: so-py39.yaml 3.30 real 2.79 user 0.49 sys 3.36 real 2.84 user 0.51 sys 3.25 real 2.74 user 0.49 sys 3.34 real 2.82 user 0.51 sys 3.29 real 2.78 user 0.51 sys 3.24 real 2.74 user 0.48 sys Timing creating environment: so-py3x.yaml 3.27 real 2.79 user 0.47 sys 3.26 real 2.78 user 0.46 sys 3.33 real 2.83 user 0.48 sys 3.28 real 2.79 user 0.47 sys 3.31 real 2.81 user 0.49 sys 3.29 real 2.81 user 0.47 sys  This indicates that when using Mamba the difference in solve time is neglible.
Question: I am trying to add type hints for data returned by plt.subplots. That works fine for plt.Axes, but I can't seem to find a solution for Figure. Any ideas what I could do? An abbreviated version of my code is: def draw_graph() -> Tuple[plt.Figure, plt.Axes]: fig, ax = plt.subplots(figsize=(14,10)) return (fig, ax)  I get the message: "Figure" is not a known member of module Pylance

Answer: With the latest Matplotlib (v3.7.1) I was able to do the following: import matplotlib.pyplot as plt import matplotlib.figure def draw_graph() -> Tuple[matplotlib.figure.Figure, plt.Axes]: fig, ax = plt.subplots(figsize=(14,10)) return (fig, ax)  I haven't tested using plt.Figure, but my IDE (i.e., VS Code) was not giving me any errors with plt.Figure.
Question: Django Channels Throw error with postman while working well with Html. I'm following Django Socket Tutorial "here's the error showing in Django". WebSocket HANDSHAKING /ws/chat/roomName/ [127.0.0.1:56504] WebSocket REJECT /ws/chat/roomName/ [127.0.0.1:56504] WebSocket DISCONNECT /ws/chat/roomName/ [127.0.0.1:56504]  "Error showing in postman when connecting to ws://127.0.0.1:8000/ws/chat/roomName/" Sec-WebSocket-Version: 13 Sec-WebSocket-Key: fSSuMD2QozIrgywqTX38/A== Connection: Upgrade Upgrade: websocket Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits Host: 127.0.0.1:8000  My Code asgi.py django_asgi_app = get_asgi_application() import digital_signage.playlist_management.routing application = ProtocolTypeRouter( { "http": django_asgi_app, "websocket": AllowedHostsOriginValidator( AuthMiddlewareStack(URLRouter(digital_signage.playlist_management.routing.websocket_urlpatterns)) ), } )  consumer.py class ChatConsumer(WebsocketConsumer): def connect(self): print("self", self) self.accept() 

Answer: Stepping through the \channels\security\websocket.py module, Channels's OriginValidator is looking for an origin header, not a hosts header, which is what Postman defaults to. Assuming your ALLOWED_HOSTS looks like: ALLOWED_HOSTS = ["localhost", "127.0.0.1"]  Then in Postman, in headers add: origin:http://127.0.0.1:8000  And that will pass the origin validator without needing to modify allowed hosts in Django or the ASGI config.
Question: I'm getting the error: SyntaxError: invalid non-printable character U+00A0 When I'm running the code below: # coding=utf-8 from PIL import Image img = Image.open("img.png")  I have tried to load different images with different formats (png, jpg, jpeg). I have tried using different versions of the Pillow library. I have also tried running it using python 2 and 3.

Answer: The problem was related to a fake space found in the third line (the empty one). It is a character that looks like a space but is actually something else which is not detected by python. By removing this character the error disappeared. The character is this:
Question: I am using NumPy 1.24.0. On running this sample code line, import numpy as np num = np.float(3)  I am getting this error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ubuntu/.local/lib/python3.8/site-packages/numpy/__init__.py", line 284, in __getattr__ raise AttributeError("module {!r} has no attribute " AttributeError: module 'numpy' has no attribute 'float'  How can I fix it?

Answer: The answer is already provided in the comments by @mattdmo and @tdelaney:  NumPy 1.20 (release notes) deprecated numpy.float, numpy.int, and similar aliases, causing them to issue a deprecation warning  NumPy 1.24 (release notes) removed these aliases altogether, causing an error when they are used   In many cases you can simply replace the deprecated NumPy types by the equivalent Python built-in type, e.g., numpy.float becomes a "plain" Python float. For detailed guidelines on how to deal with various deprecated types, have a closer look at the table and guideline in the release notes for 1.20:  ... To give a clear guideline for the vast majority of cases, for the types bool, object, str (and unicode) using the plain version is shorter and clear, and generally a good replacement. For float and complex you can use float64 and complex128 if you wish to be more explicit about the precision. For np.int a direct replacement with np.int_ or int is also good and will not change behavior, but the precision will continue to depend on the computer and operating system. If you want to be more explicit and review the current use, you have the following alternatives:  np.int64 or np.int32 to specify the precision exactly. This ensures that results cannot depend on the computer or operating system. np.int_ or int (the default), but be aware that it depends on the computer and operating system. The C types: np.cint (int), np.int_ (long), np.longlong. np.intp which is 32bit on 32bit machines 64bit on 64bit machines. This can be the best type to use for indexing.  ...  If you have dependencies that use the deprecated types, a quick workaround would be to roll back your NumPy version to 1.24 or less (as suggested in some of the other answers), while waiting for the dependency to catch up. Alternatively, you could create a patch yourself and open a pull request, or monkey patch the dependency in your own code.
Question: Currently I have the current code which runs a prompt on a model which it downloads from huggingface. from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler model_id = "stabilityai/stable-diffusion-2" # Use the Euler scheduler here instead scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler") pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler) pipe = pipe.to("mps") pipe.enable_attention_slicing() prompt = "a photo of an astronaut riding a horse on mars" pipe(prompt).images[0]  I wanted to know how can I feed a custom ckpt file to this script instead of it downloading it from stabilityAi repo?

Answer: You cannot use a ckpt file with diffusers out of the box. The ckpt file has to be converted to a diffusers friendly format. You can do that with a tool named StableTuner. Or this utility script on HuggingFace https://huggingface.co/spaces/anzorq/sd-to-diffusers
Question: Using psycopg2, I could write large results as CSV using copy_expert and a BytesIO buffer like this with pandas: copy_sql = "COPY (SELECT * FROM big_table) TO STDOUT CSV" buffer = BytesIO() cursor.copy_expert(copy_sql, buffer, size=8192) buffer.seek(0) pd.read_csv(buffer, engine="c").to_excel(self.output_file)  However, I can't figure out how to replace the buffer in copy_expert with psycopg3's new copy command. Has anyone figured out a way to do this?

Answer: The key to writing a large query to a file through psycopg3 in this fashion is to use a SpooledTemporaryFile, which will limit the amount of memory usage in Python (see max_size). Then after the CSV is written to disk, convert with pandas. from tempfile import SpooledTemporaryFile from pandas import read_csv from psycopg import connect cursor = connect([connection]).cursor() copy_sql = "COPY (SELECT * FROM stocks WHERE price > %s) TO STDOUT" price = 100 with SpooledTemporaryFile( mode="wb", max_size=65546, buffering=8192, ) as tmpfile: with cursor.copy(copy_sql, (price,)) as copy: for data in copy: tmpfile.write(data) tmpfile.seek(0) read_csv(tmpfile, engine="c").to_excel("my_spreadsheet.xlsx")
Question: I am trying to reshape the following data from wide to long format df = pd.DataFrame( { "size_Ent": { pd.Timestamp("2021-01-01 00:00:00"): 600, pd.Timestamp("2021-01-02 00:00:00"): 930, }, "size_Baci": { pd.Timestamp("2021-01-01 00:00:00"): 700, pd.Timestamp("2021-01-02 00:00:00"): 460, }, "min_area_Ent": { pd.Timestamp("2021-01-01 00:00:00"): 1240, pd.Timestamp("2021-01-02 00:00:00"): 1503, }, "min_area_Baci": { pd.Timestamp("2021-01-01 00:00:00"): 1285, pd.Timestamp("2021-01-02 00:00:00"): 953, }, } )   size_Ent size_Baci min_area_Ent min_area_Baci 2021-01-01 600 700 1240 1285 2021-01-02 930 460 1503 953  The problem is that the column names contain two different pieces of information separated by an underscore:  The property/variable that was measured (e.g. size or min_area). I'd like these to remain as column names (without duplicates). A label for the item that was measured (e.g., Ent or Baci). I'd like these labels to become the values of a new column called 'bacterium'.  Additionally, I'd like the row indexes to remain as timestamps. It should look like this:  bacterium min_area size 2021-01-01 Baci 1285 700 2021-01-01 Ent 1240 600 2021-01-02 Baci 953 460 2021-01-02 Ent 1503 930  I tried transposing the data frame with df.T but this did not give the result I want.

Answer: This can be solved in three simple steps: First, notice that your column names are actually encoding a 2x2 MultiIndex, so let's start by creating a MultiIndex from tuples. To do this, we need to first transform the existing column names into tuples. This is easy because we know they should be split at the last underscore. # Convert column names into MultiIndex, giving an informative name to the level with label data column_tuples = df.columns.str.rsplit("_", n=1) column_tuples = [tuple(c) for c in column_tuples] df.columns = pd.MultiIndex.from_tuples(column_tuples,names=[None,'bacterium'])  Next, use df.stack() to take the 'bacterium' level from the column MultiIndex and move it into a row MultiIndex. This is not quite the same as the transpose operation that you tried. df = df.stack('bacterium')  Finally, use df.reset_index() with the level argument to take the bacterium level from the row MultiIndex and make it a proper column. df = df.reset_index('bacterium')  Result:  bacterium min_area size 2021-01-01 Baci 1285 700 2021-01-01 Ent 1240 600 2021-01-02 Baci 953 460 2021-01-02 Ent 1503 930
Question: Using PyYaml import yaml yaml.full_load(StringIO('a: 16:00:00')) # {'a': 57600} yaml.full_load(StringIO('a: 09:31:00')) # {'a': '09:31:00'}  Why is there a difference in those behaviors?

Answer: Older versions of YAML supported sexagesimal (base 60) numbers, intended for use for things like times. Instead of adding additional digits (like hexadecimal uses 0-9 and A-F), it simply uses decimal numbers 0-59 separated by :s. 16:00:00 is thus equivalent to 16*(60**2) + 0*60 + 0 == 57600.  PyYAML apparently still uses this older YAML specification. 09:30:00, however, does not start with a valid decimal: a leading zero indicates an octal number, but 09 is not a valid octal number. Not being able to parse this as any kind of known number (octal, decimal, or sexagesimal), PyYAML falls back to a string. YAML can represent timestamps, but only if they consist of a date and an optional timestamp. PyYAML parses such timestamps as datetime.datetime objects, as seems reasonable. >>> yaml.full_load(StringIO('a: 2022-12-21T09:31:00')) {a: datetime.datetime(2022, 12, 21, 9, 31)}  I referenced an answer in a comment, https://stackoverflow.com/a/45165433/1126841, provided by the author of another package with does adhere to the YAML 1.2 specification, which will parse the value as a string, not a sexagesimal integer. >>> from ruamel import yaml >>> yaml.safe_load('a: 16:00:00') {'a': '16:00:00'}
Question: I am using function to count occurrences of given word in pdf using PyPDF2. While the function is running I get message in terminal: FloatObject (b'0.000000000000-14210855') invalid; use 0.0 instead  My code: def count_words(word): print() print('Counting words..') files = os.listdir('./pdfs') counted_words = [] for idx, file in enumerate(files, 1): with open(f'./pdfs/{file}', 'rb') as pdf_file: ReadPDF = PyPDF2.PdfFileReader(pdf_file, strict=False) pages = ReadPDF.numPages words_count = 0 for page in range(pages): pageObj = ReadPDF.getPage(page) data = pageObj.extract_text() words_count += sum(1 for match in re.findall(rf'\b{word}\b', data, flags=re.I)) counted_words.append(words_count) print(f'File: {idx}') return counted_words  How to get rid of this message?

Answer: See https://pypdf2.readthedocs.io/en/latest/user/suppress-warnings.html import logging logger = logging.getLogger("PyPDF2") logger.setLevel(logging.ERROR)
Question: I am reading a file using: def readFile(): file = open('Rules.txt', 'r') lines = file.readlines() for line in lines: rulesList.append(line)  rulesList: ['\n', "Rule(F1, HTTPS TCP, ['ip', 'ip'], ['www.google.ca', '8.8.8.8'], 443)\n", '\n', "Rule(F2, HTTPS TCP, ['ip', 'ip'], ['75.2.18.233'], 443)\n", '\n']  My file looks like: Rule(F1, HTTPS TCP, ['ip', 'ip'], ['www.google.ca', '8.8.8.8'], 443) Rule(F2, HTTPS TCP, ['ip', 'ip'], ['ip'], 443)  I would like to feed the values to a class I created class Rule: def __init__(self, flowNumber, protocol, port, fromIP=[], toIP=[]): self.flowNumber = flowNumber self.protocol = protocol self.port = port self.fromIP = fromIP self.toIP = toIP def __repr__(self): return f'\nRule({self.flowNumber}, {self.protocol}, {self.fromIP}, {self.toIP}, {self.port})' newRule = Rule(currentFlowNum, currentProtocol, currentPort, currentFromIP, currentToIP)  to get an output such as: [F1, HTTPS TCP, ['ip', 'ip'], ['www.google.ca', '8.8.8.8'], 443]  or be able to assign these values to a variable like: currentFlowNum = F1, currentProtocol = 'HTTPS TCP' , currentPort = 443, currentFromIP = ['ip', 'ip'], currentToIP = ['www.google.ca', '8.8.8.8']  I tried: for rule in rulesList: if rule !='\n': tmp = rule.split(',') print(tmp)  tmp: ['Rule(F1', ' HTTPS TCP', " ['ip'", " 'ip']", " ['www.google.ca'", " '8.8.8.8']", ' 443)\n'] ['Rule(F2', ' HTTPS TCP', " ['ip'", " 'ip']", " ['ip']", ' 443)\n']  Is there a way to not split the commas between [] i.e. I would like the output to look like: ['Rule(F1', ' HTTPS TCP', " ['ip','ip']", " ['www.google.ca', '8.8.8.8']", ' 443)\n'] ['Rule(F2', ' HTTPS TCP', " ['ip','ip']", " ['ip']", ' 443)\n'] 

Answer: If you have control over how the data in the file is stored and can replace the single quotes (') with double quotes (") to make the "list" structures valid JSON, you could use RegExp for this. A word of caution: unless you are absolutely sure that the format you'll be reading will largely remain the same and is rather inflexible, you're better off storing this data in a more well-established format (as mentioned in the comments) like JSON, YAML, etc. There are so many edge cases that could happen here that rolling your own parser like this objectively suboptimal. import re import json def readFile(): file = open('Rules.txt', 'r') myRules = [] for line in file.readlines(): match = re.match(r'Rule\((?P<flow_number>[^,]+),\s(?P<protocol>[^,]+),\s(?P<from_ip>\[[^\]]+\]),\s(?P<to_ip>\[[^\]]+\]),\s(?P<port>[^,)]+)\)', line) if match: myRules.append(Rule(match.group('flow_number'), match.group('protocol'), match.group('port'), json.loads(match.group('from_ip')), json.loads(match.group('to_ip')))) return myRules print(readFile()) # Returns: # [ # Rule(F1, HTTPS TCP, ['ip', 'ip'], ['www.google.ca', '8.8.8.8'], 443), # Rule(F2, HTTPS TCP, ['ip', 'ip'], ['ip'], 443)]  Repl.it | Regex101
Question: I need to downgrade my version of poetry to version 1.2.1. Currently, it's 1.2.2. >>> poetry --version Poetry (version 1.2.2)  I use the following command: >>> curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.2.1 python3 - Retrieving Poetry metadata The latest version (1.2.1) is already installed.  But I'm told that 1.2.1 is already installed. Yet the poetry version is still stuck on the original. >>> poetry --version Poetry (version 1.2.2)  The answer given here doesn't work (poetry self update@1.2.1) => The command "self" does not exist. What am I doing wrong here?

Answer: if you want to install specific version in python hereit is , pip install poetry==1.2.1  In Future just simplpe pip install 'Your Library Name'== 'Specific version'
Question: I have a script that can be run independently but sometimes will be externally invoked with parameters meant to override the ones defined in the script. I got it working using exec() (the safety of this approach is not the point here) but I don't understand why it works in a for loop and not in a comprehension list. foo = 1 bar = 2 externally_given = ['foo=10', 'bar=20'] for ext in externally_given: exec(ext) print('Exec in for loop ->', foo, bar) externally_given = ['foo=30', 'bar=40'] [exec(ext) for ext in externally_given] print('Exec in comprehension list ->', foo, bar)  Output: Exec in for loop -> 10 20 Exec in comprehension list -> 10 20  EDIT: Python version 3.10

Answer: To update global variables, let exec() have access to them by passing globals() as the second parameter: [exec(ext,globals()) for ext in externally_given] # [None, None] foo # 10 bar # 20  (Subject to all the good comments to the original post.)
Question: When I run this code on google colab. from google.cloud import aiplatform The following error occurred ImportError: cannot import name 'WKBWriter' from 'shapely.geos' (/usr/local/lib/python3.8/dist-packages/shapely/geos.py) Does anyone know how to solve this problem? I was working fine on 2022/12/16, but today it is not working.

Answer: The bug is tracked in: https://github.com/googleapis/python-aiplatform/issues/1852 The workaround is to pin shapely < 2.0.0 pip install -U google-cloud-aiplatform "shapely<2"
Question: I have a list of objects of the following type: @dataclass class Feature: name: str active: bool  and my list is: features = [Feature("name1",False), Feature("name2",False), Feature("name3",True)]  I want to get back a list with all the features but switch their active property to True. I tried to use map() like this: active_features=list(map(lambda f: f.active=True,features))  but it gives me an error expected parameter. How can this be achieved? Note I thought it was following from the example, but I guess I should have clarified. I want to do this with some short of inline method, without defining a new separate function as suggested from some of the answers, but maybe it cannot be done like this?

Answer: Reason why your logic is not working? It gives you error because the lambda function you're using is trying to modify the value of f.active, which is not allowed in a lambda function. lambda functions are allowed to only for expressions that return a value, rather than statements that perform some actions. So I think one way to do it like below- from dataclasses import dataclass import copy @dataclass class Feature: name: str active: bool features = [Feature("name1",False), Feature("name2",False), Feature("name3",True)] active_features = [] main_features = copy.deepcopy(features) for f in main_features: f.active = True active_features.append(f) print(active_features) print(features)  Output: [Feature(name='name1', active=True), Feature(name='name2', active=True), Feature(name='name3', active=True)] [Feature(name='name1', active=False), Feature(name='name2', active=False), Feature(name='name3', active=True)]
Question: {{ config ( pre_hook = before_begin("{{audit_tbl_insert(1,'stg_news_sentiment_analysis_incr') }}"), post_hook = after_commit("{{audit_tbl_update(1,'stg_news_sentiment_analysis_incr','dbt_development','news_sentiment_analysis') }}") ) }} select rd.news_id ,rd.title, rd.description, ns.sentiment from live_crawler_output_rss.rss_data rd left join live_crawler_output_rss.news_sentiment ns on rd.news_id = ns.data_id limit 10000;  This is my model in DBT which is configured with pre and post hooks which referance a macro to insert and update the audit table. my macro { % macro audit_tbl_insert (model_id_no, model_name_txt) % } {% set run_id_value = var('run_id') %} insert into {{audit_schema_name}}.{{audit_table_name}} (run_id, model_id, model_name, status, start_time, last_updated_at) values ({{run_id_value}}::bigint,{{model_id_no}}::bigint,{{model_name_txt}},'STARTED',current_timestamp,current_timestamp) {% endmacro %}  this is the first time i'm using this macro and I see the following error. Compilation Error in model stg_news_sentiment_analysis_incr (models/staging/stg_news_sentiment_analysis_incr.sql) 'audit_tbl_insert' is undefined in macro run_hooks (macros/materializations/hooks.sql) called by macro materialization_table_default (macros/materializations/models/table/table.sql) called by model stg_news_sentiment_analysis_incr (models/staging/stg_news_sentiment_analysis_incr.sql). This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps". 

Answer: Your macro's definition has too much whitespace in the braces that define the jinja block: { % macro audit_tbl_insert (model_id_no, model_name_txt) % }  Needs to be {% macro audit_tbl_insert (model_id_no, model_name_txt) %}  And then this should work just fine.