eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am reading data from csv files which has about 50 columns, few of the columns(4 to 5) contain text data with non-ASCII characters and special characters. df = spark.read.csv(path, header=True, schema=availSchema)  I am trying to remove all the non-Ascii and special characters and keep only English characters, and I tried to do it as below df = df['textcolumn'].str.encode('ascii', 'ignore').str.decode('ascii')  There are no spaces in my column name. I receive an error  TypeError: 'Column' object is not callable --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <command-1486957561378215> in <module> ----> 1 InvFilteredDF = InvFilteredDF['SearchResultDescription'].str.encode('ascii', 'ignore').str.decode('ascii') TypeError: 'Column' object is not callable  Is there an alternative to accomplish this, appreciate any help with this.

Answer: This should work. First creating a temporary example dataframe: df = spark.createDataFrame([ (0, "This is Spark"), (1, "I wish Java could use case classes"), (2, "Data science is cool"), (3, "This is ï»¿aSA") ], ["id", "words"]) df.show()  Output +---+--------------------+ | id| words| +---+--------------------+ | 0| This is Spark| | 1|I wish Java could...| | 2|Data science is ...| | 3| This is ï»¿aSA| +---+--------------------+  Now to write a UDF because those functions that you use cannot be directly performed on a column type and you will get the Column object not callable error Solution from pyspark.sql.functions import udf def ascii_ignore(x): return x.encode('ascii', 'ignore').decode('ascii') ascii_udf = udf(ascii_ignore) df.withColumn("foo", ascii_udf('words')).show()  Output +---+--------------------+--------------------+ | id| words| foo| +---+--------------------+--------------------+ | 0| This is Spark| This is Spark| | 1|I wish Java could...|I wish Java could...| | 2|Data science is ...|Data science is ...| | 3| This is ï»¿aSA| This is aSA| +---+--------------------+--------------------+
Question: I would like to use Visual Studio Code's debugger to debug my python code, but exception occurs. I use Windows 10, WSL, Debian, Python 3.7.6.  Exception has occurred: ModuleNotFoundError No module named 'flask' File "/home/kazu/test/main.py", line 2, in <module> from flask import Flask   This is python debugger console's record. pyenv shell 3.7.6 /home/kazu/.pyenv/versions/3.7.6/bin/python /home/kazu/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py --default --client --host localhost --port 52440 /home/kazu/test/main.py kazu@D:~/test$ pyenv shell 3.7.6 kazu@D~/test$ /home/kazu/.pyenv/versions/3.7.6/bin/python /home/kazu/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py --default --client --host localhost --port 52440 /home/kazu/test/main.py  However, I have already installed flask using pipenv. When I don't use debugger, there isn't module error. This is my main.py from __future__ import unicode_literals from flask import Flask from flask import render_template from flask import request from flask import send_file import os import youtube_dl app = Flask(__name__) @app.route("/", methods=['POST', 'GET']) def index(): if request.method == "POST": if os.path.exists("/tmp/output.mp4"): os.remove("/tmp/output.mp4") else: print("Can not delete the file as it doesn't exists") url = request.form['url'] ydl_opts = {'outtmpl': '/tmp/output.mp4', 'format':'bestvideo[ext=mp4]'} with youtube_dl.YoutubeDL(ydl_opts) as ydl: ydl.download([url]) return send_file("/tmp/output.mp4",as_attachment=True) else: return render_template("index.html") if __name__ == "__main__": app.run()  I searched the Internet and found that I should put my .venv folder in the project directory. So, I operated this command. export PIPENV_VENV_IN_PROJECT=1  and now my directory structure is this. . ├── main.py ├── Pipfile ├── Pipfile.lock ├── .venv └── templates └── index.html  However, I get same error message. Then, I searched the Internet again and this time I set vs code's python venv path, but I got the same error message.  Could you give me any information or suggestion? Thank you in advance. Sincerely, Kazu

Answer: If you look in the bottom-left corner of your screen you will notice you are currently running against a pyenv install of Python and not a pipenv virtual environment. If you click on the interpreter name and select the appropriate environment where you installed flask it should fix your issue.
Question: The following dataclass: from abc import ABC from collections.abc import Mapping from dataclasses import dataclass, field @dataclass(eq=True, order=True, frozen=True) class Expression(Node, ABC): def node(self): raise NotImplementedError  is used as a base class for: @dataclass(eq=True, frozen=True) class HashLiteral(Expression): pairs: Mapping[Expression, Expression] ...  Node is defined as: @dataclass(eq=True, frozen=True) class Node: def __str__(self) -> str: raise NotImplementedError  When trying to use the HashLiteral class I get the error: pairs: Mapping[Expression, Expression] TypeError: 'ABCMeta' object is not subscriptable  What is wrong with my annotation of pairs above?

Answer: You should use typing.Mapping instead of collections.abc.Mapping. typing contains many generic versions of various types, which are designed to be used in type hints. According to the mypy documentation, there are some differences between the typing classes and the collections.abc classes, but they're unclear on exactly what those differences are.
Question: I'm trying to get Uber's Ludwig to run. I get an error about there being no attribute 'random_normal'. I can reproduce the error in Python with these commands. >>> import tensorflow as tf >>> tf.reduce_sum(tf.random_normal([1000,1000])) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'tensorflow' has no attribute 'random_normal' >>> print(tf.__version__) 2.1.0 >>> print(sys.version) 3.7.5 (defaut, Oct 25 2019, 15:51:11) [GCC 7.3.0]  How can I get past this error?

Answer: It was moved to tf.random.normal (along with all the other tf.random_* functions)
Question: I'm trying to set up an Enum that will return None if the value is not found. The documentation mentions a function _missing_, but does not explain any of the details regarding the function:  _missing_ – a lookup function used when a value is not found; may be overridden  After some looking around, it seems this is a classmethod with the signature cls, value, so I tried to set it up, and its not working. >>> class G(enum.Enum): ... @classmethod ... def _missing_(cls, value): ... return None ... a = 1 ... >>> G(1) <G.a: 1> >>> G(2) Traceback (most recent call last): ... ValueError: 2 is not a valid G >>> G['b'] KeyError: 'b' >>> G.b AttributeError: b  A google search suggests that _missing_ only catches the ValueError in the call case, so the KeyError and TypeError do not surprise me, but I do not know why G(2) is raising ValueError instead of returning None.

Answer: The 2 main things that the documentation is missing regarding the _missing_ function is the signature in the question, and the fact that the return type MUST be a member of the Enum. If None is returned, then the error simply isn't silenced. This behaviour is only seen through source inspection or a different error message: >>> class G(enum.Enum): ... @classmethod ... def _missing_(cls, value): ... return "a truthy value" # I suspected that the error may have been caused by a falsey return ... a = 1 ... >>> G(2) ValueError: 2 is not a valid G During handling of the above exception, another exception occured: Traceback (most recent call last): ... TypeError: error in G._missing_: returned 'a truthy value' instead of None or a valid member  So the only way to handle this case is to have a sentinal G.none, G.null, G.missing or whatever value is most suitable.
Question: I'm building a visualization with Python. There I'd like to visualize fuel stops and the fuel costs of my car. Furthermore, car washes and their costs should be visualized as well as repairs. The fuel costs and laundry costs should have a higher bar depending on the costs. I created the visualization below to describe the concepts. How to create such a visualization with matplotlib? This is the visualization being built: 

Answer: Yes, this kind of visualization is perfectly possible with matplotlib. To store the data, numpy arrays are usually very handy. Here is some code to get you started: import matplotlib.pyplot as plt import numpy as np refuel_km = np.array([0, 505.4, 1070, 1690]) refuel_cost = np.array([40.1, 50, 63, 55]) carwash_km = np.array([302.0, 605.4, 901, 1331, 1788.2]) carwash_cost = np.array([35.0, 40.0, 35.0, 35.0, 35.0]) repair_km = np.array([788.0, 1605.4]) repair_cost = np.array([135.0, 74.5]) fig, ax = plt.subplots(figsize=(12,3)) plt.scatter(refuel_km, np.full_like(refuel_km, 0), marker='o', s=100, color='lime', edgecolors='black', zorder=3, label='refuel') plt.bar(refuel_km, refuel_cost, bottom=15, color='lime', ec='black', width=20, label='refuel cost') plt.scatter(carwash_km, np.full_like(carwash_km, 0), marker='d', s=100, color='tomato', edgecolors='black', zorder=3, label='car wash') plt.bar(carwash_km, -carwash_cost, bottom=-15, color='tomato', ec='black', width=20, label='car wash cost') plt.scatter(repair_km, np.full_like(repair_km, 0), marker='^', s=100, color='lightblue', edgecolors='black', zorder=3, label='car repair') #plt.bar(repair_km, -repair_cost, bottom=-15, color='lightblue', ec='black', width=20) ax.spines['bottom'].set_position('zero') ax.spines['top'].set_color('none') ax.spines['right'].set_color('none') ax.spines['left'].set_color('none') ax.tick_params(axis='x', length=20) ax.set_yticks([]) # turn off the yticks _, xmax = ax.get_xlim() ymin, ymax = ax.get_ylim() ax.set_xlim(-15, xmax) ax.set_ylim(ymin, ymax+25) # make room for the legend ax.text(xmax, -5, "km", ha='right', va='top', size=14) plt.legend(ncol=5, loc='upper left') plt.tight_layout() plt.show()
Question: Assume a REST API which defines a POST method on a resource /foos to create a new Foo. When creating a Foo the name of the Foo is an input parameter (present in the request body). When the server creates a Foo it assigns it an ID. This ID is returned together with the name in the REST response. I am looking for something similar to readOnly in OpenAPI. The input JSON should look like this: { "name": "bar" }  The output JSON should look like that: { "id": 123, "name": "bar" }  Is there a way to reuse the same pydantic model? Or is it necessary to use two diffent models? class FooIn(BaseModel): name: str class Foo(BaseModel): id: int name: str  I cannot find any mentions of "read only", "read-only", or "readonly" in the pydantic documentation or in the Field class code. Googling I found a post which mentions id: int = Schema(..., readonly=True)  But that seems to have no effect in my use case.

Answer: It is fine to have multiple models. You can use inheritance to reduce code repetition: from pydantic import BaseModel # Properties to receive via API create/update class Foo(BaseModel): name: str # Properties to return via API class FooDB(Foo): id: int  The documentation which is excellent btw!, goes into this more in-depth. Here is a real user model example taken from the official full stack project generator. You can see how there are multiple models to define the user schema depending on the context.
Question: What's the difference between the following two TypeVars? from typing import TypeVar, Union class A: pass class B: pass T = TypeVar("T", A, B) T = TypeVar("T", bound=Union[A, B])   I believe that in Python 3.12 this is the difference between these two bounds class Foo[T: (A, B)]: ... class Foo[T: A | B]: ...   Here's an example of something I don't get: this passes type checking... T = TypeVar("T", bound=Union[A, B]) class AA(A): pass class X(Generic[T]): pass class XA(X[A]): pass class XAA(X[AA]): pass  ...but with T = TypeVar("T", A, B), it fails with  error: Value of type variable "T" of "X" cannot be "AA"   Related: this question on the difference between Union[A, B] and TypeVar("T", A, B).

Answer: When you do T = TypeVar("T", bound=Union[A, B]), you are saying T can be bound to either Union[A, B] or any subtype of Union[A, B]. It's upper-bounded to the union. So for example, if you had a function of type def f(x: T) -> T, it would be legal to pass in values of any of the following types:  Union[A, B] (or a union of any subtypes of A and B such as Union[A, BChild]) A (or any subtype of A) B (or any subtype of B)  This is how generics behave in most programming languages: they let you impose a single upper bound.  But when you do T = TypeVar("T", A, B), you are basically saying T must be either upper-bounded by A or upper-bounded by B. That is, instead of establishing a single upper-bound, you get to establish multiple! So this means while it would be legal to pass in values of either types A or B into f, it would not be legal to pass in Union[A, B] since the union is neither upper-bounded by A nor B.  So for example, suppose you had a iterable that could contain either ints or strs. If you want this iterable to contain any arbitrary mixture of ints or strs, you only need a single upper-bound of a Union[int, str]. For example: from typing import TypeVar, Union, List, Iterable mix1: List[Union[int, str]] = [1, "a", 3] mix2: List[Union[int, str]] = [4, "x", "y"] all_ints = [1, 2, 3] all_strs = ["a", "b", "c"] T1 = TypeVar('T1', bound=Union[int, str]) def concat1(x: Iterable[T1], y: Iterable[T1]) -> List[T1]: out: List[T1] = [] out.extend(x) out.extend(y) return out # Type checks a1 = concat1(mix1, mix2) # Also type checks (though your type checker may need a hint to deduce # you really do want a union) a2: List[Union[int, str]] = concat1(all_ints, all_strs) # Also type checks a3 = concat1(all_strs, all_strs)  In contrast, if you want to enforce that the function will accept either a list of all ints or all strs but never a mixture of either, you'll need multiple upper bounds. T2 = TypeVar('T2', int, str) def concat2(x: Iterable[T2], y: Iterable[T2]) -> List[T2]: out: List[T2] = [] out.extend(x) out.extend(y) return out # Does NOT type check b1 = concat2(mix1, mix2) # Also does NOT type check b2 = concat2(all_ints, all_strs) # But this type checks b3 = concat2(all_ints, all_ints)
Question: I am trying to use Prometheus for monitoring Celery tasks for which I am relatively new and I have a problem with incrementing a counter. It's just not incrementing if I am trying to do it inside of Celery.task E.g. from celery import Celery from prometheus_client import Counter app = Celery('tasks', broker='redis://localhost') TASKS = Counter('tasks', 'Count of tasks') @app.task def add(x, y): TASKS.inc(1) return x + y  When I visit an endpoint to see which metrics are exposed I can see tasks_total, but its value doesn't change no matter how much add tasks have been executed. However, when I am trying to increment the same counter from regular function, it works.  E.g.  def dummy_add(x, y): TASKS.inc() return x + y  Could you please explain to me what I am doing wrong?

Answer: By default, Celery uses process pool for workers. This doesn't go well with Prometheus Python client and you have to use its multiprocessing mode. You might rather want to use one of existing Celery exporters (such as this one) which take a different approach. They just start its own process and listen on events from workers thus overcoming the drawbacks mentioned above.
Question: I have just started learning Spark. Currently, I am trying to perform One hot encoding on a single column from my dataframe. However I cannot import the OneHotEncoderEstimator from pyspark. I have try to import the OneHotEncoder (depacated in 3.0.0), spark can import it but it lack the transform function. Here is the output from my code below. If anyone has encountered similar problem, please help. Thank you so much for your time!!  

Answer: Your first problem is that encoder object has no 'transform' error. This is a category indexer. Before you can transform columns of object, you must train a OneHotEncoderEstimator using fit() function. In that way your encoder object will learn from data and will be able to transfer the data to encoded category vectors. Most of the category indexer models requires fit() function to learn from data itself.  so what you should do is encoder = OneHotEncoderEstimator(dropLast=False, inputCol:"AgeIndex", outputCol="AgeVec" model = encoder.fit(df) encoded = model.transform(df) encoded.show()  Also I recommend you to read documentation before starting to a project if you are new to something, documentation helps a lot. The section of spark that includes transformation operations posted here as a link.  Spark Transformation Operations  your second problem is import error, since you are using notebook I suggest you should check your notebook's environment. But your version is preview version which mostly considers the developers and tester. For starters one should always go for the latest tested release. Try to switch back to spark-2.4.4 and check the notebook's environment.
Question: I have a csv file as the given picture bellow   I'm trying to find any word that will start with letter A and G or any list that I want  but my code returns an error any Ideas what I'm doing wrong ? this is my code if len(sys.argv) == 1: print("please provide a CSV file to analys") else: fileinput = sys.argv[1] wdata = pd.read_csv(fileinput) print( list(filter(startswith("a","g"), wdata)) ) 

Answer: Use Series.str.startswith with convert list to tuple and filtering by DataFrame.loc with boolean indexing: wdata = pd.DataFrame({'words':['what','and','how','good','yes']}) L = ['a','g'] s = wdata.loc[wdata['words'].str.startswith(tuple(L)), 'words'] print (s) 1 and 3 good Name: words, dtype: object
Question: I have a dictionary in python with a pretty standard structure. I want to retrieve one value if present, and if not retrieve another value. If for some reason both values are missing I need to throw an error. Example of dicts: # Data that has been modified data_a = { "date_created": "2020-01-23T16:12:35+02:00", "date_modified": "2020-01-27T07:15:00+02:00" } # Data that has NOT been modified data_b = { "date_created": "2020-01-23T16:12:35+02:00", }  What is the best practice for this? What could be an intuitive and easily readable way to do this? The way I do this at the moment: mod_date_a = data_a.get('date_modified', data_a.get('date_created', False)) # mod_data_a = "2020-01-27T07:15:00+02:00" mod_date_b = data_b.get('date_modified', data_b.get('date_created', False)) # mod_data_b = "2020-01-23T16:12:35+02:00" if not mod_date_a or not mod_date_b: log.error('Some ERROR')  This nested get() just seems a little clumsy so I was wondering if anyone had a better solution for this.

Answer: Assuming there are no false values and you don't really need False but just anything false: mod_date_a = data_a.get('date_modified') or data_a.get('date_created')
Question: I would like to remove elements that are greater than a threshold from a list. For example, a list with elements a = [1,9,2,10,3,6]. I would like to remove all elements that are greater than 5. Return should be [1,2,3]. I tried using enumerate and pop but it doesn't work. for i,x in enumerate(a): if x > 5: a.pop(i) 

Answer: Try using a list comprehension: >>> a = [1,9,2,10,3,6] >>> [x for x in a if x <= 5] [1, 2, 3]  This says, "make a new list of x values where x comes from a but only if x is less than or equal to the threshold 5. The issue with the enumerate() and pop() approach is that it mutates the list while iterating over it -- somewhat akin to sawing-off a tree limb while your still sitting on the limb. So when (i, x) is (1, 9), the pop(i) changes a to [1,2,10,3,6], but then iteration advances to (2, 10) meaning that the value 2 never gets examined. It falls apart from there. FWIW, if you need to mutable the list in-place, just reassign it with a slice: a[:] = [x for x in a if x <= 5]  Hope this helps :-)
Question: Let's for the sake of example assume I want to subclass dict and have all keys capitalized: class capdict(dict): def __init__(self,*args,**kwds): super().__init__(*args,**kwds) mod = [(k.capitalize(),v) for k,v in super().items()] super().clear() super().update(mod) def __getitem__(self,key): return super().__getitem__(key.capitalize()) def __setitem__(self,key,value): super().__setitem__(key.capitalize(),value) def __delitem__(self,key): super().__detitem__(key.capitalize())  This works to an extent,  >>> ex = capdict(map(reversed,enumerate("abc"))) >>> ex {'A': 0, 'B': 1, 'C': 2} >>> ex['a'] 0  but, of course, only for methods I remembered to implement, for example >>> 'a' in ex False  is not the desired behavior. Now, the lazy way of filling in all the methods that can be derived from the "core" ones would be mixing in collections.abc.MutableMapping. Only, it doesn't work here. I presume because the methods in question (__contains__ in the example) are already provided by dict. Is there a way of having my cake and eating it? Some magic to let MutableMapping only see the methods I've overridden so that it reimplements the others based on those?

Answer: What you could do: This likely won't work out well (i.e. not the cleanest design), but you could inherit from MutableMapping first and then from dict second. Then MutableMapping would use whatever methods you've implemented (because they are the first in the lookup chain): >>> class D(MutableMapping, dict): def __getitem__(self, key): print(f'Intercepted a lookup for {key!r}') return dict.__getitem__(self, key) >>> d = D(x=10, y=20) >>> d.get('x', 0) Intercepted a lookup for 'x' 10 >>> d.get('z', 0) Intercepted a lookup for 'z' 0  Better way: The cleanest approach (easy to understand and test) is to just inherit from MutableMapping and then implement the required methods using a regular dict as the base data store (with composition rather than inheritance): >>> class CapitalizingDict(MutableMapping): def __init__(self, *args, **kwds): self.store = {} self.update(*args, **kwds) def __getitem__(self, key): key = key.capitalize() return self.store[key] def __setitem__(self, key, value): key = key.capitalize() self.store[key] = value def __delitem__(self, key): del self.store[key] def __len__(self): return len(self.store) def __iter__(self): return iter(self.store) def __repr__(self): return repr(self.store) >>> d = CapitalizingDict(x=10, y=20) >>> d {'X': 10, 'Y': 20} >>> d['x'] 10 >>> d.get('x', 0) 10 >>> d.get('z', 0) 0 >>> d['w'] = 30 >>> d['W'] 30
Question: I am having trouble understanding what exactly the python function gc.collect() does. Does the function only collects those objects that are collectable, to free up the space at a later time once the gc.threshold is reached? or does gc.collect() collect objects and also automatically get rid of whatever it collected, so that the space can be freed immediately after executing the code?

Answer: It depends! If called with no argument or with generation=2 as an argument, it would free the objects that are collectable. If called with generation=1 it would not clear the free lists. From the documentation:  With no arguments, run a full collection. The optional argument generation may be an integer specifying which generation to collect (from 0 to 2). A ValueError is raised if the generation number is invalid. The number of unreachable objects found is returned. The free lists maintained for a number of built-in types are cleared whenever a full collection or collection of the highest generation (2) is run. Not all items in some free lists may be freed due to the particular implementation, in particular float.
Question: After renaming a DataFrame's column(s), I get an error when merging on the new column(s): import pandas as pd df1 = pd.DataFrame({'a': [1, 2]}) df2 = pd.DataFrame({'b': [3, 1]}) df1.columns = [['b']] df1.merge(df2, on='b')   TypeError: only integer scalar arrays can be converted to a scalar index 

Answer: Replaced the code tmp.columns = [['POR','POR_PORT']] with tmp.rename(columns={'Locode':'POR', 'Port Name':'POR_PORT'}, inplace=True) and it worked.