eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to access google cloud storage as in the code below. # amazon s3 connection import s3fs as fs with fs.open("s3://mybucket/image1.jpg") as f: image = Image.open(f).convert("RGB") # Is there an equivalent code like this GCP side? with cloudstorage.open("gs://my_bucket/image1.jpg") as f: image = Image.open(f).convert("RGB") 

Answer: You're looking for gcsfs. Both s3fs and gcsfs are part of the fsspec project and have very similar APIs. import gcsfs fs = gcsfs.GCSFileSystem() with fs.open("gs://my_bucket/image1.jpg") as f: image = Image.open(f).convert("RGB")  Note that both of these can be accessed from the fsspec interface, as long as you have the underlying drivers installed, e.g.: import fsspec with fsspec.open('s3://my_s3_bucket/image1.jpg') as f: image1 = Image.open(f).convert("RGB") with fsspec.open('gs://my_gs_bucket/image1.jpg') as f: image2 = Image.open(f).convert("RGB") # fsspec handles local paths too! with fsspec.open('/Users/myname/Downloads/image1.jpg') as f: image3 = Image.open(f).convert("RGB")  fsspec is the file system handler underlying pandas and other libraries which parse cloud URLs. The reason the following "just works" is because fsspec is providing the cloud URI handling: pd.read_csv("s3://path/to/my/aws.csv") pd.read_csv("gs://path/to/my/google.csv") pd.read_csv("my/local.csv")
Question: Given the following table: group_a = {'ba':[2.0,9.4,10.8], 'bb':[4.2,7.1,3], 'bc':[8.1,9.5,6.1]} A = pd.DataFrame(group_a, index=['aa','ab','ac'])  That looks like this:  ba bb bc aa 2.0 4.2 8.1 ab 9.4 7.1 9.5 ac 10.8 3.0 6.1  How can I flatten this table so that it looks like this:  Values aa_ba 2.0 aa_bb 4.2 aa_bc 8.1 ab_ba 9.4 ab_bb 7.1 ab_bc 9.5 ac_ba 10.8 ac_bb 3.0 ac_bc 6.1 

Answer: You can use stack and rework the index: B = A.stack() B.index = B.index.map('_'.join) out = B.to_frame('Values')  output:  Values aa_ba 2.0 aa_bb 4.2 aa_bc 8.1 ab_ba 9.4 ab_bb 7.1 ab_bc 9.5 ac_ba 10.8 ac_bb 3.0 ac_bc 6.1
Question: I'm trying to install a module called "discord_slash" and when I use it in a python file it displays the error "ModuleNotFoundError: No module named 'discord_slash'". I've tried uninstalling it and installing it again but it's not working. 

Answer: You're attempting to use the legacy v3 version import of the library, available here. As of 4.0, you should be using import interactions. Based on a comment, I understand you're watching a video from 2021, if you'd like to use a similar version you can use discord-py-slash-command 3.0.3 which is the latest release of that branch. You can install that like this: pip install discord-py-interactions=3.0.3 I'd highly recommend you go ahead and find a newer video guide or read their Quickstart documentation here so you can have the up to date library and features.
Question: I have 30 instances of the Room objects, i.e. 30 rows in the database table. In Python code I have Room.objects.all().delete(). I see that Django ORM translated it into the following PostgreSQL query: DELETE FROM "app_name_room" WHERE "app_name_room"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30). Why doesn't the Django ORM use a more parsimonious DELETE FROM app_name_room query? Is there any way to switch to it and avoid listing all IDs?

Answer: Interesting question. It got me thinking so I went a little deeper. The main reason could be that using DELETE FROM app_name_room doesn't take care of CASCADE delete However, answering your question  Is there any way to switch to it and avoid listing all IDs?  You can do this using the private method _raw_delete. For instance: objects_to_delete = Foo.objects.all() objects_to_delete._raw_delete(objects_to_delete.db)  This will execute the following query: DELETE FROM "objects_to_delete"  PS: According to the function docstring:  Delete objects found from the given queryset in single direct SQL query. No signals are sent and there is no protection for cascades.
Question: I'd like to print three dots in a row (to form an ellipsis), but print() prints blank. print("one moment...") one moment... print("...") print("..") .. print("...abc...") abc... print("\u2026") …  What's happening here? Why is "..." parsed in an exceptional way? I am using ipython in PyCharm.

Answer: Update 2024-07-31 According to the YouTrack issue (archive) this issue has been fixed and is available in builds: 242.6184, 241.16163, 241.17011.12, 242.10180.17.  Original Post Looks like this is a known issue with Pycharm where its interactive console removes the leading three periods from a print statement. Here’s the ticket tracking this issue.  A possible workaround for now is defining something like: def iprint(obj): if (s:=str(obj)).startswith("..."): print(" "+s) else: print(s)  which looks like: >>> iprint("...ymmv") ...ymmv
Question: I know polars does not support index by design, so df.filter(expr).index isn't an option, another way I can think of is by adding a new column before applying any filters, not sure if this is an optimal way for doing so in polars df.with_columns(pl.Series('index', range(len(df))).filter(expr).index 

Answer: Use with_row_index(): df = pl.DataFrame([pl.Series("a", [5, 9, 6]), pl.Series("b", [8, 3, 4])])  In [20]: df.with_row_index() Out[20]: shape: (3, 3) ┌────────┬─────┬─────┐ │ index ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 │ ╞════════╪═════╪═════╡ │ 0 ┆ 5 ┆ 8 │ │ 1 ┆ 9 ┆ 3 │ │ 2 ┆ 6 ┆ 4 │ └────────┴─────┴─────┘  # Start from 1 instead of 0. In [21]: df.with_row_index(offset=1) Out[21]: shape: (3, 3) ┌────────┬─────┬─────┐ │ index ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 │ ╞════════╪═════╪═════╡ │ 1 ┆ 5 ┆ 8 │ │ 2 ┆ 9 ┆ 3 │ │ 3 ┆ 6 ┆ 4 │ └────────┴─────┴─────┘  # Start from 1 and call column "my_index". In [22]: df.with_row_index(name="my_index", offset=1) Out[22]: shape: (3, 3) ┌──────────┬─────┬─────┐ │ my_index ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 │ ╞══════════╪═════╪═════╡ │ 1 ┆ 5 ┆ 8 │ │ 2 ┆ 9 ┆ 3 │ │ 3 ┆ 6 ┆ 4 │ └──────────┴─────┴─────┘
Question: (M1 MBA 2020, MacOS 12.3.1) So inside of Vs Code, when I select my interpreter as Python 3.8.9 from my usr/local/bin Tkinter it runs as I want it to.  Here is the running code for reference.  The problem arises when I am trying to use the Global Python 3.8.9 interpreter (usr/bin/python3). When the code runs, the application ends up looking like this.  Additionally, when I run the code the terminal reads the following: DEPRECATION WARNING: The system version of Tk is deprecated and may be removed in a future release. Please don't rely on it. Set TK_SILENCE_DEPRECATION=1 to suppress this warning.  How is it possible for me to fix this error? Or update my global Tkinter version without straying away from Python 3.8.9. Furthermore if any more info is needed I'll be happy to provide, sorry I'm new to this stuff 😅 Packages used in the app: tkinter, Pillow, tkmacosx One last thing, when I get rid of all mentions of the package Tkmacosx, the app appears like this: 

Answer: If you have Homebrew installed, you can update tk with: brew uninstall tcl-tk --devel brew install tcl-tk  Which is the recommended option Then you may need to add export PATH="/usr/local/opt/tcl-tk/bin:$PATH" to your .zshrc file: If you are using a zsh terminal:  Use: echo "# For tkinter export PATH=\"/usr/local/opt/tcl-tk/bin:\$PATH\"" >> ~/.zshrc  source ~/.zshrc  Or if you are using a bash terminal: echo "# For tkinter export PATH=\"/usr/local/opt/tcl-tk/bin:\$PATH\"" >> ~/.bashrc  source ~/.bashrc  If the steps above didn't work, you may need to install another package (@goker): brew install python-tk@3.10   Homebrew Reference Python's offical tk upgrade docs
Question: When reading source code of fastapi, this line make me fuzzy: from starlette.testclient import TestClient as TestClient  Why not just: from starlette.testclient import TestClient?

Answer: From the point of view of executable code, there is absolutely no difference in terms of the Python bytecode being generated by the two different code examples (using Python 3.9): >>> dis.dis('from starlette.testclient import TestClient as TestClient') 1 0 LOAD_CONST 0 (0) 2 LOAD_CONST 1 (('TestClient',)) 4 IMPORT_NAME 0 (starlette.testclient) 6 IMPORT_FROM 1 (TestClient) 8 STORE_NAME 1 (TestClient) 10 POP_TOP 12 LOAD_CONST 2 (None) 14 RETURN_VALUE >>> dis.dis('from starlette.testclient import TestClient') 1 0 LOAD_CONST 0 (0) 2 LOAD_CONST 1 (('TestClient',)) 4 IMPORT_NAME 0 (starlette.testclient) 6 IMPORT_FROM 1 (TestClient) 8 STORE_NAME 1 (TestClient) 10 POP_TOP 12 LOAD_CONST 2 (None) 14 RETURN_VALUE  As shown, they are exactly identical. (Related question asked before existence of PEP-0484). However, the comment by Graham501617 noted how modern type hinting validators (such as mypy) accept this particular syntax to denote the re-export of that imported name (the other being the __all__, which thankfully they did end up correctly supporting as that has been a standard syntax to denote symbols to (re-)export since Python 2). Specifically, as per the description of Stub Files in the referenced PEP 0484, quote:   Modules and variables imported into the stub are not considered exported from the stub unless the import uses the import ... as ... form or the equivalent from ... import ... as ... form. (UPDATE: To clarify, the intention here is that only names imported using the form X as X will be exported, i.e. the name before and after as must be the same.)   Looking at git blame for the relevant file in the packages pointed to this commit (direct link to relevant diff for the file) indicated that this particular type hinting issue was being addressed (as part of this issue) to ensure mypy will treat those imported names as re-export, thus allowing the usage of the --no-implicit-reexport flag (which --strict has likely implicitly enabled). This particular re-export syntax however is very much strict about the "name before and after as must be the same", the syntax referred in the related question (i.e. import foo.bar as bar) can be found in recommendation by certain modern packages (e.g. PyTorch recommends the use of import torch.nn as nn, as discussed in this question), does not in fact allow the bar (or nn) symbol be re-exported from the current module as foo.bar is not the same as bar (likewise torch.nn is not the same as nn, as the whole set of tokens torch.nn is evaluated instead of the just the final identifier after the final .).
Question: I am trying to learn how to send an email using python. All the tutorials on the web that I have read explain how to do it using Gmail. But, from 30/05/2022 (despite the fact that everybody is free to do whatever he wants with his account) Google has a new policy that states:  To help keep your account secure, starting May 30, 2022, Google will no longer support the use of third-party apps or devices that only ask for your username and password for you. Sign in to your Google account.  Source: https://support.google.com/accounts/answer/6010255 And we get:  So my question is there any other way to send an email using python, (including email accounts belonging to an other company)? Here is my function to send an email: def send_email_fct(filename, filepath, fromaddr, mdpfrom, toaddr): """" filename: file name to be sent with extension filepath: file path of the file to be sent fromaddr: sender email address mdpfrom: password of sender email address toaddr: receiver email address""" msg = MIMEMultipart() # instance of MIMEMultipart msg['From'] = fromaddr msg['To'] = toaddr msg['Subject'] = "data file" body_email = "Body_of_the_mail" msg.attach(MIMEText(body_email, 'plain')) attachment = open(filepath, 'rb') # open the file to be sent p = MIMEBase('application', 'octet-stream') # instance of MIMEBase p.set_payload(attachment.read()) encoders.encode_base64(p) p.add_header('Content-Disposition', "attachment; filename= %s" % filename) msg.attach(p) # attach the instance 'p' to instance 'msg' s = smtplib.SMTP('smtp.gmail.com', 587) # SMTP s.starttls() s.login(fromaddr, mdpfrom) text = msg.as_string() s.sendmail(from_email_addr, toaddr, text) # sending the email s.quit() # terminating the session  And I get this error: smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8 https://support.google.com/mail/?p=BadCredentials c12-20020aa7d60c000000b0042be14040c1sm2612116edr.86 - gsmtp')  To fix this problem, I think that the only line that need to be change is this one: s = smtplib.SMTP('smtp.gmail.com', 587)  If you know by what I can change it or if you see any other error, it will help me a lot! :-)

Answer: Here is a more precise answer with all the main steps. I hope it will help other people.  Log in into your email account: https://myaccount.google.com  Then go to the security part    Be sure that you have turn on two steps verification and click on "App password." As of 9th July 2023, this might not be visible, but is reachable directly via myaccount.google.com/apppasswords   After select email and the corresponding device    It will generate a password that will look like this; it is this password that you have to use in your python script.    This password will appear here; in this place, you will have the option to erase it (so it will be no longer be useful to connect to your email account).   Hope it will help other people!
Question: I have a class that mocks database functionality which does not subclass Mock or MagicMock because it defines its own __init__() method: class DatabaseMock(): def __init__(self, host=None): self.host = host self.x = {} # other methods that mutate x  There is a function I want to test that makes an API call to the real database, so I patched it out: from unittest.mock import patch class TestFunctions(): def test_function(self): with patch("path.to.database.call", DatabaseMock) as mock: result = function_i_am_testing() assert mock.x == result  There is a field of the DatabaseMock called x, but in the patch context, mock.x returns an AttributeError. This leads to me believe mock is not really an instance of DatabaseMock(). Also, I had tried making x a class level object which makes x visible, but its state would persist through separate test calls which I do not want. What is mock and how can I reference the mocked object instance in the context?

Answer: I have figured out the issue. When patch is given a class, it will return a class, not an object instance of that class. So in my example, mock is not a DataBaseMock object instance, but a reference to the class. This is why class level variables are visible, but not object instance fields. In order to get my desired functionality, I did this: from unittest.mock import patch class TestFunctions(): def test_function(self): with patch("path.to.database.call") as mock: mock.return_value = DataBaseMock() result = function_i_am_testing() assert mock.return_value.x == result  Now, mock is a MagicMock object, whose return value is the object I need.
Question: Given this custom TypedDict TMyDict: class TMyDict(TypedDict, total=False): prop_a: int prop_b: int  This is ok: def get_my_dict_works() -> TMyDict: return { 'prop_a': 0, 'prop_b': 1 }  But this don't: def get_my_dict_fail() -> TMyDict: d= { 'prop_a': 0, 'prop_b': 1 } return d  Error message is:  Expression of type "dict[str, Unknown]" cannot be assigned to return type "TMyDict" "dict[str, Unknown]" is incompatible with "TMyDict"  And it works if I add the type annotation when assigning var: def get_my_dict_fix() -> TMyDict: d: TMyDict = { 'prop_a': 0, 'prop_b': 1 } return d  Why?

Answer: Your type checker determined the type of d before you used it In order to be performant and to keep type inference simple, type checkers typically only scan the code once and try to determine the type of a variable the moment it is assigned. It saw the line d = { 'prop_a': 0, 'prop_b': 1 }  and assumed that d: dict[str, int] or maybe d: dict[str, Unknown]. If the type checker tried to accept your code, it would have to jump back and check all before uses of d the moment d is used as TMyDict. This could be very costly and as just adding the annotation fixes the error, it is not supported in most type checkers. Also, the code is cleaner if the user can directly see that d is supposed to be an TMyDict instead of just any dict.
Question: I have following code import unicodecsv CSV_PARAMS = dict(delimiter=",", quotechar='"', lineterminator='\n') unireader = unicodecsv.reader(open('sample.csv', 'rb'), **CSV_PARAMS) for line in unireader: print(line)  and it prints ['\ufeff"003', 'word one"'] ['003,word two'] ['003,word three']  The CSV looks like this "003,word one" "003,word two" "003,word three"  I am unable to figure out why the first row has \ufeff (which is i believe a file marker). Moreover, there is " at the beginning of first row. The CSV file is comign from client so i can't dictate them how to save a file etc. Looking to fix my code so that it can handle encoding. Note: I have already tried passing encoding='utf8' to CSV_PARAMS and it didn't solve the problem

Answer: encoding='utf-8-sig' will remove the UTF-8-encoded BOM (byte order mark) used as a UTF-8 signature in some files: import unicodecsv with open('sample.csv','rb') as f: r = unicodecsv.reader(f, encoding='utf-8-sig') for line in r: print(line)  Output: ['003,word one'] ['003,word two'] ['003,word three']  But why are you using the third-party unicodecsv with Python 3? The built-in csv module handles Unicode correctly: import csv # Note, newline='' is a documented requirement for the csv module # for reading and writing CSV files. with open('sample.csv', encoding='utf-8-sig', newline='') as f: r = csv.reader(f) for line in r: print(line)
Question: I have the following code that I am using to try to chunk response from a http.client.HTTPSConnection get request to an API (please note that the response is gzip encoded:  connection = http.client.HTTPSConnection(api, context = ssl._create_unverified_context()) connection.request('GET', api_url, headers = auth) response = connection.getresponse() while chunk := response.read(20): data = gzip.decompress(chunk) data = json.loads(chunk) print(data)  This always gives out an error that it is not a gzipped file (b'\xe5\x9d'). Not sure how I can chunk data and still achieve what I am trying to do here. Basically, I am chunking so that I don't have to load the entire response in memory. Please note I can't use any other libraries like requests, urllib etc.

Answer: The most probable reason for that is, the response your received is indeed not a gzipped file. I notice that in your code, you pass a variable called auth. Typically, a server won't send you a compressed response if you don't specify in the request headers that you can accept it. If there is only auth-related keys in your headers like your variable name suggests, you won't receive a gzipped response. First, make sure you have 'Accept-Encoding': 'gzip' in your headers. Going forward, you will face another problem:  Basically, I am chunking so that I don't have to load the entire response in memory.  gzip.decompress will expect a complete file, so you would need to reconstruct it and load it entirely in memory before doing that, which would undermine the whole point of chunking the response. Trying to decompress a part of a gzip with gzip.decompress will most likely give you an EOFError saying something like Compressed file ended before the end-of-stream marker was reached. I don't know if you can manage that directly with the gzip library, but I know how to do it with zlib. You will also need to convert your chunk to a file-like object, you can do that with io.BytesIO. I see you have very strong constraints on libraries, but zlib and io are part of the python default, so hopefully you have them available. Here is a rework of your code that should help you going on: import http import ssl import gzip import zlib from io import BytesIO # your variables here api = 'your_api_host' api_url = 'your_api_endpoint' auth = {'AuhtKeys': 'auth_values'} # add the gzip header auth['Accept-Encoding'] = 'gzip' # prepare decompressing object decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS) connection = http.client.HTTPSConnection(api, context = ssl._create_unverified_context()) connection.request('GET', api_url, headers = auth) response = connection.getresponse() while chunk := response.read(20): data = decompressor.decompress(BytesIO(chunk).read()) print(data)
Question: Context I created two data classes to handle table metadata. TableMetadata apply to any kind of tables, while RestTableMetadata contains information relevant for data extracted from REST apis @dataclass class TableMetadata: """ - entity: business entity represented by the table - origin: path / query / url from which data withdrawn - id: field to be used as ID (unique) - historicity: full, delta - upload: should the table be uploaded """ entity: str origin: str view: str id: str = None historicity: str = "full" upload: bool = True columns: list = field(default_factory=list) @dataclass class RestTableMetadata(TableMetadata): """ - method: HTTP method to be used - payloadpath: portion of the response payload to use to build the dataframe """ method: str payloadpath: str = None  Problem Because of inheritance, method (without default values) comes after columns, resulting in the following Pylance error: Fields without default values cannot appear after fields with default values I'm looking for a way to fix it without overriding __init__ (if there is such a way). I also noticed a method called __init_subclass__ (This method is called when a class is subclassed.) that might affect how RestTableMetadata.__init__ and other subclasses is generated.

Answer: Here is a working solution for python > 3.10 @dataclass(kw_only=True) class TableMetadata: """ - entity: business entity represented by the table - origin: path / query / url from which data withdrawn - id: field to be used as ID (unique) - historicity: full, delta - upload: should the table be uploaded """ entity: str origin: str view: str id: str = None historicity: str = "full" upload: bool = True columns: list = field(default_factory=list) @dataclass(kw_only=True) class RestTableMetadata(TableMetadata): """ - method: HTTP method to be used - payloadpath: portion of the response payload to use to build the dataframe """ method: str payloadpath: str = None
Question: I'm new to python. I'm using a firebase firestore database for this project. After entering the Admission_No, I want to retrieve all the data in the relevant document such as name, grade, phone. I tried to write a program for it. But I failed. Please help me to complete my project. Thank you all. The Code might be full of mistakes. Please don't mind it and fix them for me. Sample Data  Here is my Code .py import firebase_admin from firebase_admin import credentials from firebase_admin import firestore cred = credentials.Certificate("cred.json") firebase_admin.initialize_app(cred) db = firestore.client() data = { 'Admission_No': input('enter ad_no : ') } query_ad = db.collection(u'Users').where(u"Admission_No", u"==", data["Admission_No"]).get() get_data = db.collection(u'Users').where(u"Name", u"==", data["Name"]).get() if query_ad: print('Exist') print(get_data) else: print("Doesn't Exist") 

Answer: In this case, I tried many ways. But finally, I realized we should provide the correct Document ID for retrieve all fields of data from firestore. Getting Data from Firestore - Method given in firestore documentation doc_ref = db.collection(u'cities').document(u'SF') doc = doc_ref.get() if doc.exists: print(f'Document data: {doc.to_dict()}') else: print(u'No such document!')  In my case I tried with this Code and it was succeed. u_name = input('Enter User Name : ') ad_no = input('Enter Admission Number : ') doc_name = ad_no + " - " + u_name doc_ref = self.db.collection(u'Users').document(doc_name) doc = doc_ref.get() if doc.exists: Full_Name = str(doc.to_dict()['Full_Name']) Admission_No = str(doc.to_dict()['Admission_No']) Grade = str(doc.to_dict()['Grade']) Phone_Number = str(doc.to_dict()['Phone_Number']) print(Full_Name, Admission_No, Grade, Phone_Number) else: print('No such document!')  Thank you everyone who helped me with this issue. (@RJC)