eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to obtain the "Where from" extended file attribute which is located on the "get info" context-menu of a file in MacOS. Example When right-clicking on the file and displaying the info it shows the this metadata. The highlighted part in the image below shows the information I want to obtain (the link of the website where the file was downloaded from).  I want to use this Mac-specific function using Python. I thought of using OS tools but couldn't figure out any.

Answer: TL;DR: Get the extended attribute like MacOS's "Where from" by e.g. pip-install pyxattr and use xattr.getxattr("file.pdf", "com.apple.metadata:kMDItemWhereFroms"). Extended Attributes on files These extended file attributes like your "Where From" in MacOS (since 10.4) store metadata not interpreted by the filesystem. They exist for different operating systems. using the command-line You can also query them on the command-line with tools like:  exiftool:  exiftool -MDItemWhereFroms -MDItemTitle -MDItemAuthors -MDItemDownloadedDate /path/to/file   xattr (apparently MacOS also uses a Python-script)  xattr -p -l -x /path/to/file  On MacOS many attributes are displayed in property-list format, thus use -x option to obtain hexadecimal output. using Python Ture Pålsson pointed out the missing link keywords. Such common and appropriate terms are helpful to search Python Package Index (PyPi): Search PyPi by keywords: extend file attributes, meta data:  xattr pyxattr osxmetadata, requires Python 3.7+, MacOS only  For example to list and get attributes use (adapted from pyxattr's official docs) import xattr xattr.listxattr("file.pdf") # ['user.mime_type', 'com.apple.metadata:kMDItemWhereFroms'] xattr.getxattr("file.pdf", "user.mime_type") # 'text/plain' xattr.getxattr("file.pdf", "com.apple.metadata:kMDItemWhereFroms") # ['https://example.com/downloads/file.pdf']  However you will have to convert the MacOS specific metadata which is stored in plist format, e.g. using plistlib. File metadata on MacOS Mac OS X 10.4 (Tiger) introduced Spotlight a system for extracting (or harvesting), storing, indexing, and querying metadata. It provides an integrated system-wide service for searching and indexing. This metadata is stored as extended file attributes having keys prefixed with com.apple.metadata:. The "Where from" attribute for example has the key com.apple.metadata:kMDItemWhereFroms. using Python Use osxmetadata to use similar functionality like in MacOS's md* utils: from osxmetadata import OSXMetaData filename = 'file.pdf' meta = OSXMetaData(filename) # get and print "Where from" list, downloaded date, title print(meta.wherefroms, meta.downloadeddate, meta.title)  See also  MacIssues (2014): How to look up file metadata in OS X OSXDaily (2018): How to View & Remove Extended Attributes from a File on Mac OS Ask Different: filesystem - What all file metadata is available in macOS? Query Spotlight for a range of dates via PyObjC Mac OS X : add a custom meta data field to any file
Question: I have to check whether url exists or not using python. I am trying to use requests.get(url) but it is taking alot of time as the file starts downloading as soon as get is hit. I don't want the file to be downloaded for checking the url validity. Can this be achieved using python ?

Answer: Something like the below. See HTTP head for more info. import requests urls = ['https://www.google.com','https://www.google.com/you_can_not_find_me'] for idx,url in enumerate(urls,1): r = requests.head(url) if r.status_code == 200: print(f'{idx}) {url} was found') else: print(f'{idx}) {url} was NOT found')  output 1) https://www.google.com was found 2) https://www.google.com/you_can_not_find_me was NOT found
Question: I would like to use the following sdk in my python project -> https://github.com/LBank-exchange/lbank-api-sdk-v2. It has sdk's for 3 languages (I just want the python one). I tried to install it using the command: pip install git+https://github.com/LBank-exchange/lbank-api-sdk-v2.git#egg=lbank which gave the error does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

Answer: Looks like the developer didn't bother to package it properly. If it was me using it, I would fork it on GH, add the setup.py and use the fork. Maybe a good exercise for you? Meanwhile, to just get it to work, in your project "root": git clone https://github.com/LBank-exchange/lbank-api-sdk-v2.git ln -s lbank-api-sdk-v2/python-sdk-api/LBank ./LBank  Then in your code just import LBank. This will leave the cloned repo untouched (so you can git pull to update it later) and just link the module directory to the root. Alternatively you can just include the api directory in sys.path for imports to work.
Question: Basic question about python f-strings, but couldn't find out the answer: how to force sign display of a float or integer number? i.e. what f-string makes 3 displayed as +3?

Answer: From Docs:  Format Specification Mini-Language(Emphasis mine):      Option Meaning     '+' indicates that a sign should be used for both positive as well as negative numbers.   '-' indicates that a sign should be used only for negative numbers (this is the default behavior).     Example from docs: >>> '{:+f}; {:+f}'.format(3.14, -3.14) # show it always '+3.140000; -3.140000' >>> '{:-f}; {:-f}'.format(3.14, -3.14) # show only the minus -- same as '{:f}; {:f}' '3.140000; -3.140000' >>> '{:+} {:+}'.format(10, -10) '+10 -10'  Above examples using f-strings: >>> f'{3.14:+f}; {-3.14:+f}' '+3.140000; -3.140000' >>> f'{3.14:-f}; {-3.14:-f}' '3.140000; -3.140000' >>> f'{10:+} {-10:+}' '+10 -10'  One caveat while printing 0 as 0 is neither positive nor negative. In python, +0 = -0 = 0. >>> f'{0:+} {-0:+}' '+0 +0' >>> f'{0.0:+} {-0.0:+}' '+0.0 -0.0'  0.0 and -0.0 are different objects1.  0 in Computer Science(Emphasis mine):   In some computer hardware signed number representations, zero has two distinct representations, a positive one grouped with the positive numbers and a negative one grouped with the negatives; this kind of dual representation is known as signed zero, with the latter form sometimes called negative zero.  Update: From Python 3.11 and above, allows negative floating point zero as positive zero.  The 'z' option coerces negative zero floating-point values to positive zero after rounding to the format precision. This option is only valid for floating-point presentation types.  Example from PEP682:  >>> x = -.00001 >>> f'{x:z.1f}' '0.0' >>> x = decimal.Decimal('-.00001') >>> '{:+z.1f}'.format(x) '+0.0'     1. Negative 0 in Python. Also check out Signed Zero (-0)
Question: I found https://github.com/python-gitlab/python-gitlab, but I was unable to understand the examples in the doc.

Answer: That's right there are no tests we can find in the doc. Here's a basic answer for your question. If you would like a complete working script, I have attached it here: https://github.com/hubshashwat/common_scripts/blob/main/automation_to_create_push_merge_in_gitlab/usecase_gitlab_python.py Breaking down the steps below: Create an authkey for you: Follow the steps here: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html Create a gitlab server instance of your project server = gitlab.Gitlab('https://gitlab.example.com', private_token=YOUR_API_TOKEN) project = server.projects.get(PROJECT_ID)  Create a branch using: branch = project.branches.create( {"branch": branch_name, "ref": project.default_branch} )  Upload a file using: project.files.create( { "file_path": file_name, "branch": branch.name, "content": "data to be written", "encoding": "text", # or 'base64'; useful for binary files "author_email": AUTHOR_EMAIL, # Optional "author_name": AUTHOR_NAME, # Optional "commit_message": "Create file", } )  Create a merge request using: project.mergerequests.create( { "source_branch": branch.name, "target_branch": project.default_branch, "title": "merge request title", } )
Question: This code: a = [1, 2, 3] print(*a, a.pop(0))  Python 3.8 prints 2 3 1 (does the pop before unpacking). Python 3.9 prints 1 2 3 1 (does the pop after unpacking). What caused the change? I didn't find it in the changelog. Edit: Not just in function calls but also for example in a list display: a = [1, 2, 3] b = [*a, a.pop(0)] print(b)  Prints [2, 3, 1] vs [1, 2, 3, 1]. And Expression lists says "The expressions are evaluated from left to right" (that's the link to Python 3.8 documentation), so I'd expect the unpacking expression to happen first.

Answer: I suspect this may have been an accident, though I prefer the new behavior. The new behavior is a consequence of a change to how the bytecode for * arguments works. The change is in the changelog under Python 3.9.0 alpha 3:  bpo-39320: Replace four complex bytecodes for building sequences with three simpler ones. The following four bytecodes have been removed:  BUILD_LIST_UNPACK BUILD_TUPLE_UNPACK BUILD_SET_UNPACK BUILD_TUPLE_UNPACK_WITH_CALL  The following three bytecodes have been added:  LIST_TO_TUPLE LIST_EXTEND SET_UPDATE   On Python 3.8, the bytecode for f(*a, a.pop()) looks like this:  1 0 LOAD_NAME 0 (f) 2 LOAD_NAME 1 (a) 4 LOAD_NAME 1 (a) 6 LOAD_METHOD 2 (pop) 8 CALL_METHOD 0 10 BUILD_TUPLE 1 12 BUILD_TUPLE_UNPACK_WITH_CALL 2 14 CALL_FUNCTION_EX 0 16 RETURN_VALUE  while on 3.9, it looks like this:  1 0 LOAD_NAME 0 (f) 2 BUILD_LIST 0 4 LOAD_NAME 1 (a) 6 LIST_EXTEND 1 8 LOAD_NAME 1 (a) 10 LOAD_METHOD 2 (pop) 12 CALL_METHOD 0 14 LIST_APPEND 1 16 LIST_TO_TUPLE 18 CALL_FUNCTION_EX 0 20 RETURN_VALUE  In the old bytecode, the code pushes a and (a.pop(),) onto the stack, then unpacks those two iterables into a tuple. In the new bytecode, the code pushes a list onto the stack, then does l.extend(a) and l.append(a.pop()), then calls tuple(l). This change has the effect of shifting the unpacking of a to before the pop call, but this doesn't seem to have been deliberate. Looking at bpo-39320, the intent was to simplify the bytecode instructions, not to change the behavior, and the bpo thread has no discussion of behavior changes.  A related change affects ** unpacking, listed in the changelog under Python 3.9.0 alpha 4:  bpo-39320: Replace two complex bytecodes for building dicts with two simpler ones. The new bytecodes DICT_MERGE and DICT_UPDATE have been added The old bytecodes BUILD_MAP_UNPACK and BUILD_MAP_UNPACK_WITH_CALL have been removed.  So d = {'a': 1} f(**d, b=d.pop('a'))  also changed behavior between Python 3.8 and 3.9.
Question: i am trying to add data labels values on top of my histogram to try to show the frequency visibly. This is my code now but unsure how to code up to put the value ontop: plt.figure(figsize=(15,10)) plt.hist(df['Age'], edgecolor='white', label='d') plt.xlabel("Age") plt.ylabel("Number of Patients") plt.title = ('Age Distrubtion')  I was wondering if anyone knows the code to do this: 

Answer: You can use the new bar_label() function using the bars returned by plt.hist(). Here is an example: from matplotlib import pyplot as plt import pandas as pd import numpy as np df = pd.DataFrame({'Age': np.random.randint(20, 60, 200)}) plt.figure(figsize=(15, 10)) values, bins, bars = plt.hist(df['Age'], edgecolor='white') plt.xlabel("Age") plt.ylabel("Number of Patients") plt.title('Age Distrubtion') plt.bar_label(bars, fontsize=20, color='navy') plt.margins(x=0.01, y=0.1) plt.show()   PS: As the age is discrete distribution, it is recommended to explicitly set the bin boundaries, e.g. plt.hist(df['Age'], bins=np.arange(19.999, 60, 5)).
Question: What do the squiggly lines represent in the image? The actual error the flags up when I hover my mouse over the squiggly line is: Import "pyspark.sql.functions" could not be resolvedPylance I'm not sure what that means, but I'm getting the error for almost all functions in Visual Studio Code. How can I resolve it?    

Answer: I was with the same error as yours. Visual Studio Code usually has a "recommended" interpreter, but sometimes it won't help you out with what you need. So,  I changed the Interpreter (Ctrl + Shift + P in Visual Studio Code). Look for "Python: Select Interpreter. Choose the one who contains the name "Conda"  And that's how the magic happens.
Question: I am trying to get a random n number of users from a set of unique users. Here is what I have so far users = set() random_users = random.sample((users), num_of_user)  This works well but it is giving me a deprecated warning. What should I be using instead? random.choice doesn't work with sets UPDATE I am trying to get reactions on a post and want them to be unique which is why I used a set. Would it be better to stick with a list for this? users = set() for reaction in msg.reactions: async for user in reaction.users(): users.add(user) 

Answer: Convert your set to a list.  by using the list function: random_users = random.choices(list(users),k=num_of_user)   by using * operator to unpack your set or dict: random_users = random.choices([*users],k=num_of_user)    Solution 1. is 3 char longer than the 2., but solution 1. is more literal - to me. It is not guaranteed that you will get the same result for the list order through different executions, python versions and platforms, therefore you may end up with different random result, despite a careful random number generator initialization. To resolve this issue, order your list. You can also store the users in a list, and make the elements unique later with set, and then convert again to a list. For this, a common way is to convert your list to a set, and back to a list again.
Question: The below code fails mypy with error: Overloaded function signatures 1 and 2 overlap with incompatible return types. @overload def test_overload(x: str) -> str: ... @overload def test_overload(x: object) -> int: ... def test_overload(x) -> Union[str, int]: if isinstance(x, str): return x else: return 1  What I'm trying to express is: "This function takes an arbitrary Python object. If that object is a string, it returns a string. If it is any other type, it returns an integer. Note this particular example is contrived to represent the general case. Is it possible to express this with overloads?

Answer: At the moment (Python 3.10, mypy 0.961) there is no way to express any object except one. But you could use ignoring type: ignore[misc] for excepted types. And they must precede the more general variant, because for @overload order is matter: from typing import overload, Union @overload def test_overload(x: str) -> str: # type: ignore[misc] ... @overload def test_overload(x: object) -> int: ... def test_overload(x) -> Union[str, int]: if isinstance(x, str): return x else: return 1 reveal_type(test_overload("string")) # Revealed type is "builtins.str" reveal_type(test_overload(object())) # Revealed type is "builtins.int"
Question: When we fetch data using the DB API 2.0 cur.execute("select * from db.table") we get a cursor which seems like a generator object of list of tuples. Whereas in pymongo, when we fetch we get it as list of dictionaries. I wanted to achieve something like this. Instead of fetching list of tuples, I wanted list of dictionaries or a named tuple. I believe from an efficiency point of view it makes sense, since the schema is already defined so no need to send it for every record. Currently the workaround I am using is: cur.execute("select * from db.table") columns = cur.columns_with_types data = cur.fetchall() df = pd.DataFrame(data,columns=[tuple[0] for tuple in columns]) data_reqd = df.to_dict('records')  This methods fairs poorly when query returns a lot of data. Workaround 1: Use fetchmany(size=block_size) but it doesn't seem like an elegant way to do things. Workaround 2: This seems like a much better way to handle things. cur.execute("select * from db.table") columns = cur.columns_with_types for tup in cur: row = dict(zip(columns, tup)) # use row  Any good way to handle this? Any improvements to the question are appreciated.

Answer: You can alternatively create a Client and call its query_dataframe method. import clickhouse_driver as ch ch_client = ch.Client(host='localhost') df = ch_client.query_dataframe('select * from db.table') records = df.to_dict('records')
Question: Suppose I have a dataframe as below: df = pd.DataFrame({'a':[1,2,3,4],'b':[2,3,4,5],'c':[3,4,5,6],'d':[5,3,2,4]})  I want to check if elements in column d exist elsewhere in its corresponding row. So the outcome I want is [False, True, False, True]  Towards that end, I used df.apply(lambda x: x['d'] in x[['a','b','c']], axis=1)  but this is somehow giving me [False, False, False, False].

Answer: Try: out = (df[['a','b','c']].T==df['d']).any()  Output: 0 False 1 True 2 False 3 True dtype: bool
Question: How to change colors in decision tree plot using sklearn.tree.plot_tree without using graphviz as in this question: Changing colors for decision tree plot created using export graphviz? plt.figure(figsize=[21, 6]) ax1 = plt.subplot(121) ax2 = plt.subplot(122) ax1.plot(X[:, 0][y == 0], X[:, 1][y == 0], "bo") ax1.plot(X[:, 0][y == 1], X[:, 1][y == 1], "g^") ax1.contourf(xx, yy, pred.reshape(xx.shape), cmap=matplotlib.colors.ListedColormap(['b', 'g']), alpha=0.25) ax1.set_title(title) plot_tree(tree_clf, feature_names=["X", "y"], class_names=["blue", "green"], filled=True, rounded=True)  

Answer: Many matplotlib functions follow the color cycler to assign default colors, but that doesn't seem to apply here. The following approach loops through the generated annotation texts (artists) and the clf tree structure to assign colors depending on the majority class and the impurity (gini). Note that we can't use alpha, as a transparent background would show parts of arrows that are usually hidden. from matplotlib import pyplot as plt from matplotlib.colors import ListedColormap, to_rgb import numpy as np from sklearn import tree X = np.random.rand(50, 2) * np.r_[100, 50] y = X[:, 0] - X[:, 1] > 20 clf = tree.DecisionTreeClassifier(random_state=2021) clf = clf.fit(X, y) fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=[21, 6]) colors = ['crimson', 'dodgerblue'] ax1.plot(X[:, 0][y == 0], X[:, 1][y == 0], "o", color=colors[0]) ax1.plot(X[:, 0][y == 1], X[:, 1][y == 1], "^", color=colors[1]) xx, yy = np.meshgrid(np.linspace(X[:, 0].min(), X[:, 0].max(), 100), np.linspace(X[:, 1].min(), X[:, 1].max(), 100)) pred = clf.predict(np.c_[(xx.ravel(), yy.ravel())]) ax1.contourf(xx, yy, pred.reshape(xx.shape), cmap=ListedColormap(colors), alpha=0.25) # ax2.set_prop_cycle(mpl.cycler(color=colors)) # doesn't seem to work artists = tree.plot_tree(clf, feature_names=["X", "y"], class_names=colors, filled=True, rounded=True, ax=ax2) for artist, impurity, value in zip(artists, clf.tree_.impurity, clf.tree_.value): # let the max value decide the color; whiten the color depending on impurity (gini) r, g, b = to_rgb(colors[np.argmax(value)]) f = impurity * 2 # for N colors: f = impurity * N/(N-1) if N>1 else 0 artist.get_bbox_patch().set_facecolor((f + (1-f)*r, f + (1-f)*g, f + (1-f)*b)) artist.get_bbox_patch().set_edgecolor('black') plt.tight_layout() plt.show()
Question: OS: monterey macOSv12.0.1 python venv: 3.9.9 requirements.in # To update requirements.txt, run: # # pip-compile requirements.in # # To install in localhost, run: # # pip-sync requirements.txt # django==3.2.10 # https://www.djangoproject.com/ psycopg2-binary==2.9.2 # https://github.com/psycopg/psycopg2  After i turn on venv, then i type pip-compile requirements.in then i get a bunch of errors about pg_config not found This is my asciinema https://asciinema.org/a/sl9MqmrayLAR3rRxEul4mYaxw I have tried env LDFLAGS='-L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' pip-compile requirements.in but same. Please advise.

Answer: I appreciate the other 2 answers from @Vishnudev and @cetver 🙏 But I tried to install postgresql using brew install and it took a very long time and I still cannot complete after 20 mins. I figured this out eventually after much googling Here are my tech specs of my situation:  monterey 12.1.0 Apple silicon zsh  Concepts Conceptually, what I did is :  install openssl, turn on all the exports associated with that, then install libpq, turn on all the exports associated with that, then turn on python venv  Here are my steps I took. It's entirely possible not all steps are needed. But I have limited time so here they are. Steps  brew install openssl put the following in .zshrc export PATH="/opt/homebrew/opt/openssl@1.1/bin:$PATH" put the following in .zshenv  export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib" export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include" export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@1.1/lib/pkgconfig"   source ~/.zshrc brew install libpq put the following in .zshrc export PATH="/opt/homebrew/opt/libpq/bin:$PATH" put the following in .zshenv  export LDFLAGS="-L/opt/homebrew/opt/libpq/lib" export CPPFLAGS="-I/opt/homebrew/opt/libpq/include" export PKG_CONFIG_PATH="/opt/homebrew/opt/libpq/lib/pkgconfig"   source ~/.zshenv Turn on venv Now the installing of psycopg2-binary should work  Links that help me:  the one abt openssl the one abt libpq
Question: Consider the following case: class Base: ... class Sub(Base): ... def get_base_instance(*args) -> Base: ... def do_something_with_sub(instance: Sub): ...  Let's say I'm calling get_base_instance in a context where I kow it will return a Sub instance - maybe based on what args I'm passing. Now I want to pass the returned instance to do_something_with_sub: sub_instance = get_base_instance(*args) do_something_with_sub(sub_instance)  The problem is that my IDE complains about passing a Base instance to a method that only accepts a Sub instance. I think I remember from other programming languages that I would just cast the returned instance to Sub. How do I solve the problem in Python? Conditionally throw an exception based on the return type, or is there a better way?

Answer: I think you were on the right track when you thought about it in terms of casting. We could use cast from typing to stop the IDE complaining. For example: from typing import cast class Base: pass class Sub(Base): pass def get_base_instance(*args) -> Base: return Sub() def do_something_with_sub(instance: Sub): print(instance) sub_instance = cast(Sub, get_base_instance()) do_something_with_sub(sub_instance)
Question: I have 2 columns in pandas, with data that looks like this. code fx category AXD AXDG.R cat1 AXF AXDG_e.FE cat1 333 333.R cat1 ....  There are other categories but I am only interested in cat1. I want to combine everything from the code column, and everything after the . in the fx column and replace the code column with the new combination without affecting the other rows. code fx category AXD.R AXDG.R cat1 AXF.FE AXDG_e.FE cat1 333.R 333.R cat1 .....  Here is my code, I think I have to use regex but I'm not sure how to combine it in this way. df.loc[df['category']== 'cat1', 'code'] = df[df['category'] == 'cat1']['code'].str.replace(r'[a-z](?=\.)', '', regex=True).str.replace(r'_?(?=\.)','', regex=True).str.replace(r'G(?=\.)', '', regex=True)  I'm not sure how to select the second column also. Any help would be greatly appreciated.

Answer:  There are other categories but I am only interested in cat1  You can use str.split with series.where to add the extention for cat1: df['code'] = (df['code'].astype(str).add("."+df['fx'].str.split(".").str[-1]) .where(df['category'].eq("cat1"),df['code']))   print(df) code fx category 0 AXD.R AXDG.R cat1 1 AXF.FE AXDG_e.FE cat1 2 333.R 333.R cat1