eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a python application that uses the python logging library for some time now for printing messages both on the screen and on time rotating files and works fine. The logging configuration is as follows: import logging from logging.handlers import TimedRotatingFileHandler logging.basicConfig(level=logging.INFO if debug is not True else logging.DEBUG, format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', datefmt='%Y-%m-%d %H:%M:%S', handlers=[ TimedRotatingFileHandler(log_filename, when='midnight', interval=1), logging.StreamHandler() ] )  My problem is that PyCharm keeps highlighting the logging.basicConfig part of the configuration with the following warning:  Unexpected argument(s)  Possible callees:  basicConfig(*, filename: Optional[str]=..., filemode: str=..., format: str=..., datefmt: Optional[str]=..., level: Union[int, str, None]=..., stream: IO[str]=...) basicConfig()  Inspection info: Reports discrepancies between declared parameters and actual arguments, as well as incorrect arguments (e.g. duplicate named arguments) and incorrect argument order. Decorators are analyzed, too.  And it goes out only if I remove the handlers=[...] part of the code.  Did the basicConfig's arguments change on a specific version? If yes, what is the proposed way to achieve the same thing? I'm using python 3.6 and pycharm 2020.1 (but had the same warning for at least the past 3 updates)

Answer: This issue is reported on PyCharm bug tracker at https://youtrack.jetbrains.com/issue/PY-39762 . In short: the new keyword arguments of basicConfig in Python 3, like handler, are not recognized. That issue also mentions a workaround:  Put a caret on basicConfig - Right Click - Go to - Declaration or Usages - Click on a star on the left (on a gutter) - logging/__init__.pyi should be opened - Annotate all basicConfig definitions with @overload.  I tested it and it worked. In my case PyCharm no longer complains about force=True argument.  Did the basicConfig's arguments change on a specific version?  You can always check the docs for that: https://docs.python.org/3/library/logging.html#logging.basicConfig :  Changed in version 3.2: The style argument was added. Changed in version 3.3: The handlers argument was added. Additional checks were added to catch situations where incompatible arguments are specified (e.g. handlers together with stream or filename, or stream together with filename). Changed in version 3.8: The force argument was added.
Question: I have a plotly graph of the EUR/JPY exchange rate across a few months in 15 minute time intervals, so as a result, there is no data from friday evenings to sunday evenings. Here is a portion of the data, note the skip in the index (type: DatetimeIndex) over the weekend:  Plotting this data in plotly results in a gap over the missing dates Using the dataframe above:  import plotly.graph_objs as go candlesticks = go.Candlestick(x=data.index, open=data['Open'], high=data['High'], low=data['Low'], close=data['Close']) fig = go.Figure(layout=cf_layout) fig.add_trace(trace=candlesticks) fig.show()  Ouput:  As you can see, there are gaps where the missing dates are. One solution I've found online is to change the index to text using: data.index = data.index.strftime("%d-%m-%Y %H:%M:%S")  and plotting it again, which admittedly does work, but has it's own problem. The x-axis labels look atrocious:  I would like to produce a graph that plots a graph like in the second plot where there are no gaps, but the x-axis is displayed like as it is on the first graph. Or at least displayed in a much more concise and responsive format, as close to the first graph as possible. Thank you in advance for any help!

Answer: Even if some dates are missing in your dataset, plotly interprets your dates as date values, and shows even missing dates on your timeline. One solution is to grab the first and last dates, build a complete timeline, find out which dates are missing in your original dataset, and include those dates in: fig.update_xaxes(rangebreaks=[dict(values=dt_breaks)])  This will turn this figure:  Into this:  Complete code: import plotly.graph_objects as go from datetime import datetime import pandas as pd import numpy as np # sample data df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv') # remove some dates to build a similar case as in the question df = df.drop(df.index[75:110]) df = df.drop(df.index[210:250]) df = df.drop(df.index[460:480]) # build complete timepline from start date to end date dt_all = pd.date_range(start=df['Date'].iloc[0],end=df['Date'].iloc[-1]) # retrieve the dates that ARE in the original datset dt_obs = [d.strftime("%Y-%m-%d") for d in pd.to_datetime(df['Date'])] # define dates with missing values dt_breaks = [d for d in dt_all.strftime("%Y-%m-%d").tolist() if not d in dt_obs] # make fiuge fig = go.Figure(data=[go.Candlestick(x=df['Date'], open=df['AAPL.Open'], high=df['AAPL.High'], low=df['AAPL.Low'], close=df['AAPL.Close']) ]) # hide dates with no values fig.update_xaxes(rangebreaks=[dict(values=dt_breaks)]) fig.update_layout(yaxis_title='AAPL Stock') fig.show()
Question: So I have been able to successfully install mplfinance with pip and when I import it alone I receive no error. Though when I do: from mplfinance import candlestick_ohlc I get the error ImportError: cannot import name 'candlestick_ohlc' from 'mplfinance' I have checked command prompt again, and it says it has successfully installed mplfinance. Why am I receiving this error?

Answer: So from what I understand the Matplotlib for finance has changed so that: To access the old API with the new mplfinance package installed, change statments from: from mpl_finance import  to:  from mplfinance.original_flavor import candlestick_ohlc  and then it should work fine.
Question: I have trained a CNN in Matlab 2019b that classifies images between three classes. When this CNN was tested in Matlab it was functioning fine and only took 10-15 seconds to classify an image. I used the exportONNXNetwork function in Maltab so that I can implement my CNN in Tensorflow. This is the code I am using to use the ONNX file in python: import onnx from onnx_tf.backend import prepare import numpy as np from PIL import Image onnx_model = onnx.load('trainednet.onnx') tf_rep = prepare(onnx_model) filepath = 'filepath.png' img = Image.open(filepath).resize((224,224)).convert("RGB") img = array(img).transpose((2,0,1)) img = np.expand_dims(img, 0) img = img.astype(np.uint8) probabilities = tf_rep.run(img) print(probabilities)  When trying to use this code to classify the same test set, it seems to be classifying the images correctly but it is very slow and freezes my computer as it reaches high memory usages of up to 95+% at some points. I also noticed in the command prompt while classifying it prints this: 2020-04-18 18:26:39.214286: W tensorflow/core/grappler/optimizers/meta_optimizer.cc:530] constant_folding failed: Deadline exceeded: constant_folding exceeded deadline., time = 486776.938ms.  Is there any way I can make this python code classify faster?

Answer: In this case, it appears that the Grapper optimization suite has encountered some kind of infinite loop or memory leak. I would recommend filing an issue against the Github repo. It's challenging to debug why constant folding is taking so long, but you may have better performance using the ONNX TensorRT backend as compared to the TensorFlow backend. It achieves better performance as compared to the TensorFlow backend on Nvidia GPUs while compiling typical graphs more quickly. Constant folding usually doesn't provide large speedups for well optimized models. import onnx import onnx_tensorrt.backend as backend import numpy as np model = onnx.load("trainednet.onnx'") engine = backend.prepare(model, device='CUDA:1') filepath = 'filepath.png' img = Image.open(filepath).resize((224,224)).convert("RGB") img = array(img).transpose((2,0,1)) img = np.expand_dims(img, 0) img = img.astype(np.uint8) output_data = engine.run(img)[0] print(output_data)
Question: I am running Superset via Docker. I enabled the Email Report feature and tried it:  However, I only receive the test email report. I don't receive any emails after. This is my CeleryConfig in superset_config.py: class CeleryConfig(object): BROKER_URL = 'sqla+postgresql://superset:superset@db:5432/superset' CELERY_IMPORTS = ( 'superset.sql_lab', 'superset.tasks', ) CELERY_RESULT_BACKEND = 'db+postgresql://superset:superset@db:5432/superset' CELERYD_LOG_LEVEL = 'DEBUG' CELERYD_PREFETCH_MULTIPLIER = 10 CELERY_ACKS_LATE = True CELERY_ANNOTATIONS = { 'sql_lab.get_sql_results': { 'rate_limit': '100/s', }, 'email_reports.send': { 'rate_limit': '1/s', 'time_limit': 120, 'soft_time_limit': 150, 'ignore_result': True, }, } CELERYBEAT_SCHEDULE = { 'email_reports.schedule_hourly': { 'task': 'email_reports.schedule_hourly', 'schedule': crontab(minute=1, hour='*'), }, }  The documentation says I need to run the celery worker and beat.  celery worker --app=superset.tasks.celery_app:app --pool=prefork -O fair -c 4 celery beat --app=superset.tasks.celery_app:app  I added them to the 'docker-compose.yml': superset-worker: build: *superset-build command: > sh -c "celery worker --app=superset.tasks.celery_app:app -Ofair -f /app/celery_worker.log && celery beat --app=superset.tasks.celery_app:app -f /app/celery_beat.log" env_file: docker/.env restart: unless-stopped depends_on: *superset-depends-on volumes: *superset-volumes  Celery Worker is indeed working when sending the first email. The log file is also visible. However, the celery beat seems to not be functioning. There is also no 'celery_beat.log' created. If you'd like a deeper insight, here's the commit with the full implementation of the functionality. How do I correctly configure celery beat? How can I debug this?

Answer: I managed to solve it by altering the CeleryConfig implementation, and adding a beat service to 'docker-compose.yml' New CeleryConfig class in 'superset_config.py': REDIS_HOST = get_env_variable("REDIS_HOST") REDIS_PORT = get_env_variable("REDIS_PORT") class CeleryConfig(object): BROKER_URL = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT) CELERY_IMPORTS = ( 'superset.sql_lab', 'superset.tasks', ) CELERY_RESULT_BACKEND = "redis://%s:%s/1" % (REDIS_HOST, REDIS_PORT) CELERY_ANNOTATIONS = { 'sql_lab.get_sql_results': { 'rate_limit': '100/s', }, 'email_reports.send': { 'rate_limit': '1/s', 'time_limit': 120, 'soft_time_limit': 150, 'ignore_result': True, }, } CELERY_TASK_PROTOCOL = 1 CELERYBEAT_SCHEDULE = { 'email_reports.schedule_hourly': { 'task': 'email_reports.schedule_hourly', 'schedule': crontab(minute='1', hour='*'), }, }  Changes in 'docker-compose.yml':  superset-worker: build: *superset-build command: ["celery", "worker", "--app=superset.tasks.celery_app:app", "-Ofair"] env_file: docker/.env restart: unless-stopped depends_on: *superset-depends-on volumes: *superset-volumes superset-beat: build: *superset-build command: ["celery", "beat", "--app=superset.tasks.celery_app:app", "--pidfile=", "-f", "/app/celery_beat.log"] env_file: docker/.env restart: unless-stopped depends_on: *superset-depends-on volumes: *superset-volumes
Question: Let's assume we want to create a family of classes which are different implementations or specializations of an overarching concept. Let's assume there is a plausible default implementation for some derived properties. We'd want to put this into a base class class Math_Set_Base: @property def size(self): return len(self.elements)  So a subclass will automatically be able to count its elements in this rather silly example class Concrete_Math_Set(Math_Set_Base): def __init__(self,*elements): self.elements = elements Concrete_Math_Set(1,2,3).size # 3  But what if a subclass doesn't want to use this default? This does not work: import math class Square_Integers_Below(Math_Set_Base): def __init__(self,cap): self.size = int(math.sqrt(cap)) Square_Integers_Below(7) # Traceback (most recent call last): # File "<stdin>", line 1, in <module> # File "<stdin>", line 3, in __init__ # AttributeError: can't set attribute  I realize there are ways to override a property with a property, but I'd like to avoid that. Because the purpose of the base class is to make life as easy as possible for its user, not to add bloat by imposing a (from the subclass's narrow point of view) convoluted and superfluous access method. Can it be done? If not what's the next best solution?

Answer: A property is a data descriptor which takes precedence over an instance attribute with the same name. You could define a non-data descriptor with a unique __get__() method: an instance attribute takes precedence over the non-data descriptor with the same name, see the docs. The problem here is that the non_data_property defined below is for computation purpose only (you can't define a setter or a deleter) but it seems to be the case in your example. import math class non_data_property: def __init__(self, fget): self.__doc__ = fget.__doc__ self.fget = fget def __get__(self, obj, cls): if obj is None: return self return self.fget(obj) class Math_Set_Base: @non_data_property def size(self, *elements): return len(self.elements) class Concrete_Math_Set(Math_Set_Base): def __init__(self, *elements): self.elements = elements class Square_Integers_Below(Math_Set_Base): def __init__(self, cap): self.size = int(math.sqrt(cap)) print(Concrete_Math_Set(1, 2, 3).size) # 3 print(Square_Integers_Below(1).size) # 1 print(Square_Integers_Below(4).size) # 2 print(Square_Integers_Below(9).size) # 3  However this assumes that you have access to the base class in order to make this changes.
Question: Is there a convenient one-liner to generate a list of numbers and their negative counterparts in Python? For example, say I want to generate a list with the numbers 6 to 9 and -6 to -9. My current approach is: l = [x for x in range(6,10)] l += [-x for x in l]  A simple "one-liner" would be: l = [x for x in range(6,10)] + [y for y in range(-9, -5)]  However, generating two lists and then joining them together seems inconvenient.

Answer: I am unsure if order matters, but you could create a tuple and unpack it in a list comprehension. nums = [y for x in range(6,10) for y in (x,-x)] print(nums) [6, -6, 7, -7, 8, -8, 9, -9]
Question: I noticed that a lot of dataloaders use PIL to load and transform images, e.g. the dataset builders in torchvision.datasets.folder. My question is: why use PIL? You would need to do an np.asarray operation before turning it into a tensor. OpenCV seems to load it directly as a numpy array, and is faster too. One reason I can think of is because PIL has a rich transforms library, but I feel like several of those transforms can be quickly implemented.

Answer: There is a discussion about adding OpenCV as one of possible backends in torchvision PR. In summary, some reasons provided:  OpenCV2 loads images in BGR format which would require wrapper class to handle changing to RGB internally or format of loaded images backend dependent This in turn would lead to code duplication in functional transforms in torchvision many of which use PIL operations (as transformations to support multiple backends would be pretty convoluted) OpenCV loads images as np.array, it's not really easier to do transformations on arrays Different representation might lead to hard to catch by users bugs  PyTorch's modelzoo is dependent on RGB format as well and they would like to have it easily supported Doesn't play well with Python's multiprocessing (but it's no-issue as it was an issue for Python 2)  To be honest I don't see much movement towards this idea as there exists albumentations which uses OpenCV and can be integrated with PyTorch rather smoothly. A little off-topic, but one can choose faster backend via torchvision.set_image_backend to Intel's accimage. Also Pillow-SIMD can be used as a drop-in replacement for PIL (it is supposedly faster and recommended by fastai project). When it comes to performance benchmarks they do not seem too reliable and it's not that easy to tell AFAIK.
Question: I am exploring the Jupiter notebook for Python. While calling this method "Access OutbreakLocation data" I get this exception in Python 3.6: getting AttributeError: module 'pandas' has no attribute 'json_normalize' Any ideas how can we fix this issue?

Answer: Make sure to update to Pandas 1.0.3. Pandas prior to version 1 doesn't have json_normalize.
Question: I have an AWS Lambda function which queries API and creates a dataframe, I want to write this file to an S3 bucket, I am using: import pandas as pd import s3fs df.to_csv('s3.console.aws.amazon.com/s3/buckets/info/test.csv', index=False)  I am getting an error:  No such file or directory: 's3.console.aws.amazon.com/s3/buckets/info/test.csv'  But that directory exists, because I am reading files from there. What is the problem here? I've read the previous files like this: s3_client = boto3.client('s3') s3_client.download_file('info', 'secrets.json', '/tmp/secrets.json')  How can I upload the whole dataframe to an S3 bucket?

Answer: You can use boto3 package also for storing data to S3: from io import StringIO # python3 (or BytesIO for python2) import boto3 bucket = 'info' # already created on S3 csv_buffer = StringIO() df.to_csv(csv_buffer) s3_resource = boto3.resource('s3') s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue())
Question: If i have three points P1, P2, P3 with their coordinates(x,y) P1(x,y) and P3(x,y) are coordinate of line(start, end) and P3 is a point need to be projected. how can i find the coordinate of point r(x,y) which is projection of P3 over P1 and P2 

Answer: This solution extends to points with any geometric dimensions (2D, 3D, 4D, ...). It assumes all points are one dimensional numpy arrays (or two dimensional with one dimension shape 1). I am not sure if you require the projection to fall onto line segment or the extension of segment so I include both. You can pick whichever fits your question the best:  #distance between p1 and p2 l2 = np.sum((p1-p2)**2) if l2 == 0: print('p1 and p2 are the same points') #The line extending the segment is parameterized as p1 + t (p2 - p1). #The projection falls where t = [(p3-p1) . (p2-p1)] / |p2-p1|^2 #if you need the point to project on line extention connecting p1 and p2 t = np.sum((p3 - p1) * (p2 - p1)) / l2 #if you need to ignore if p3 does not project onto line segment if t > 1 or t < 0: print('p3 does not project onto p1-p2 line segment') #if you need the point to project on line segment between p1 and p2 or closest point of the line segment t = max(0, min(1, np.sum((p3 - p1) * (p2 - p1)) / l2)) projection = p1 + t * (p2 - p1)
Question: I have a dataclass with a field template of type Enum. When using the asdict function it converts my dataclass to a dictionary. Is it possible to use the value attribute of FoobarEnum to return the string value instead of the Enum object? My initial idea was to use the dict_factory=dict parameter of the asdict function and provide my own factory but I couldn't figure out how to do this. from dataclasses import dataclass, asdict from enum import Enum @dataclass class Foobar: name: str template: "FoobarEnum" class FoobarEnum(Enum): FIRST = "foobar" SECOND = "baz" foobar = Foobar(name="John", template=FoobarEnum.FIRST) print(asdict(foobar))  Current output: {'name': 'John', 'template': <FoobarEnum.FIRST: 'foobar'>}  Goal: {'name': 'John', 'template': 'foobar'} 

Answer: This can't be done with standard library except maybe by some metaclass enum hack I'm not aware of. Enum.name and Enum.value are builtin and not supposed to be changed. The approach of using the dataclass default_factory isn't going to work either. Because default_factory is called to produce default values for the dataclass members, not to customize access to members. You can either have the Enum member or the Enum.value as a dataclass member, and that's what asdict() will return.  If you want to keep an Enum member -not just the Enum.value- as a dataclass member, and have a function converting it to dictionary that returns the Enum.value instead of the Enum member, the correct way to do it is implementing your own method to return the dataclass as a dictionary. from dataclasses import dataclass from enum import Enum class FoobarEnum(Enum): FIRST = "foobar" SECOND = "baz" @dataclass class Foobar: name: str template: FoobarEnum def as_dict(self): return { 'name': self.name, 'template': self.template.value } # Testing. print(Foobar(name="John", template=FoobarEnum.FIRST).as_dict()) # {'name': 'John', 'template': 'foobar'}
Question: I am trying to convert a dictionary: data_dict = {'t1': '1', 't2': '2', 't3': '3'} into a dataframe: key | value| ---------------- t1 1 t2 2 t3 3  To do that, I tried: schema = StructType([StructField("key", StringType(), True), StructField("value", StringType(), True)]) ddf = spark.createDataFrame(data_dict, schema)  But I got the below error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/Cellar/apache-spark/2.4.5/libexec/python/pyspark/sql/session.py", line 748, in createDataFrame rdd, schema = self._createFromLocal(map(prepare, data), schema) File "/usr/local/Cellar/apache-spark/2.4.5/libexec/python/pyspark/sql/session.py", line 413, in _createFromLocal data = list(data) File "/usr/local/Cellar/apache-spark/2.4.5/libexec/python/pyspark/sql/session.py", line 730, in prepare verify_func(obj) File "/usr/local/Cellar/apache-spark/2.4.5/libexec/python/pyspark/sql/types.py", line 1389, in verify verify_value(obj) File "/usr/local/Cellar/apache-spark/2.4.5/libexec/python/pyspark/sql/types.py", line 1377, in verify_struct % (obj, type(obj)))) TypeError: StructType can not accept object 't1' in type <class 'str'>  So I tried this without specifying any schema but just the column datatypes: ddf = spark.createDataFrame(data_dict, StringType() & ddf = spark.createDataFrame(data_dict, StringType(), StringType()) But both result in a dataframe with one column which is key of the dictionary as below: +-----+ |value| +-----+ |t1 | |t2 | |t3 | +-----+  Could anyone let me know how to convert a dictionary into a spark dataframe in PySpark ?

Answer: You can use data_dict.items() to list key/value pairs: spark.createDataFrame(data_dict.items()).show()  Which prints +---+---+ | _1| _2| +---+---+ | t1| 1| | t2| 2| | t3| 3| +---+---+  Of course, you can specify your schema: spark.createDataFrame(data_dict.items(), schema=StructType(fields=[ StructField("key", StringType()), StructField("value", StringType())])).show()  Resulting in +---+-----+ |key|value| +---+-----+ | t1| 1| | t2| 2| | t3| 3| +---+-----+