eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here are 3 numpy arrays with some numbers: import numpy as np arr_a = np.array([[0.1, 0.2, 0.3, 0.4], [0.2, 0.3, 0.4, 0.5], [0.3, 0.4, 0.5, 0.6], [0.4, 0.5, 0.6, 0.7], [0.5, 0.6, 0.7, 0.8], [0.6, 0.7, 0.8, 0.9]]) arr_b = np.array([[0.15, 0.25, 0.35, 0.45], [0.35, 0.45, 0.55, 0.65], [0.55, 0.65, 0.75, 0.85], [0.75, 0.85, 0.95, 1.05], [0.95, 1.05, 1.15, 1.25], [1.15, 1.25, 1.35, 1.45]]) arr_c = np.array([[0.3, 0.6, 0.9, 1.2], [0.6, 0.9, 1.2, 1.5], [0.9, 1.2, 1.5, 1.8], [1.2, 1.5, 1.8, 2.1], [1.5, 1.8, 2.1, 2.4], [1.8, 2.1, 2.4, 2.7]])  Each array has a shape of (6, 4). Where rows=person and columns=time(seconds), consider each row to represent a unique person and each column to represent acceleration associated with that person for that particular moment in time. I would like to create a function called calc_acc_change which computes the change in acceleration according to a given lag value. I would like this function to take in an array and a lag value (default=2) which satisfies the following formula: acc_change(t) = acc(t) - acc(t- lag), where t=time. And I would like the output to remain an array. I've started my function as follow, but I don't know how to complete it: def calc_acc_change(array, lag=2): ... return acc_change  To test the function works, please input arr_a, arr_b and arr_c into the function and print the output. Any help is much appreciated :)

Answer: Slice your input array based on the given lag and then subtract the sliced arrays. import numpy as np def calc_acc_change(array, lag=2): acc_change = np.zeros_like(array) acc_change[:, lag:] = array[:, lag:] - array[:, :-lag] return acc_change arr_a = np.array([[0.1, 0.2, 0.3, 0.4], [0.2, 0.3, 0.4, 0.5], [0.3, 0.4, 0.5, 0.6], [0.4, 0.5, 0.6, 0.7], [0.5, 0.6, 0.7, 0.8], [0.6, 0.7, 0.8, 0.9]]) arr_b = np.array([[0.15, 0.25, 0.35, 0.45], [0.35, 0.45, 0.55, 0.65], [0.55, 0.65, 0.75, 0.85], [0.75, 0.85, 0.95, 1.05], [0.95, 1.05, 1.15, 1.25], [1.15, 1.25, 1.35, 1.45]]) arr_c = np.array([[0.3, 0.6, 0.9, 1.2], [0.6, 0.9, 1.2, 1.5], [0.9, 1.2, 1.5, 1.8], [1.2, 1.5, 1.8, 2.1], [1.5, 1.8, 2.1, 2.4], [1.8, 2.1, 2.4, 2.7]]) print("Array A:\n", calc_acc_change(arr_a)) print("Array B:\n", calc_acc_change(arr_b)) print("Array C:\n", calc_acc_change(arr_c))
Question: Let's say I have this code snippet: class A: def __init__(self, p1=1, p2=None, p3="test"): self.p1 = p1 self.p2 = p2 self.p3 = p3 a1 = A() a2 = A(p1=2, p3="blah")  What I want to do is to somehow retrieve ONLY the named parameters that are passed into the constructor, not including the optional parameters with default values. To clarify, this does NOT mean to use inspect to return the list of all parameters. So for example for a1 this would be {} since the constructor is called with no parameters, and for a2 this would be {"p1": 2, "p3": "blah"} since the constructor is called with p1 and p3. I've tried googling this and it seems that this isn't really possible, or at least I haven't found a way to do this. inspect does not work because I want to do this for each individual instantiation. using locals() within the constructor also doesn't work because it includes the optional parameters as well even if they aren't explicitly passed in.

Answer: You can intercept the arguments in __new__ like this: class A: def __init__(self, p1=1, p2=None, p3="test"): self.p1 = p1 self.p2 = p2 self.p3 = p3 def __new__(cls, *args, **kwargs): print(f"{kwargs = }") return super().__new__(cls) >>> a1 = A() kwargs = {} >>> a2 = A(p1=2, p3="blah") kwargs = {'p1': 2, 'p3': 'blah'}
Question: I have a graph like this:  Instead of the days being on top of the symbol, I was wondering if there is a way I can add this annotation between the lines? From one dot to another. I apologize if in case, this is a possible duplicate. This is my expected output:  This is my current code for the annotation:  fig.add_annotation( go.layout.Annotation( x=row["order_date"], y=row["sales"], text=f"{row['time_diff']} days", showarrow=False, align='center', yanchor='auto', yshift=10, textangle=-0 )  Any suggestions on how can I do this?

Answer: What you can do is create a new dataframe for your annotations by ordering your original df by the order_date, then take the average datetime and average sales between consecutive rows to determine where to place the annotations. To obtain the text, you can take the difference in time between consecutive orders. Your dataframe for annotations would look something like this:  time_diff sales order_date 0 5 days 5.0 2019-08-27 12:00:00 1 11 days 5.0 2019-09-04 12:00:00 2 0 days 7.5 2019-09-10 00:00:00 3 0 days 15.0 2019-09-10 00:00:00 4 5 days 20.0 2019-09-12 12:00:00 5 0 days 17.5 2019-09-15 00:00:00 6 139 days 15.0 2019-11-23 12:00:00 7 NaT NaN NaT  And here is a figure I made using some sample data similar to yours: import pandas as pd import plotly.express as px import plotly.graph_objects as go ## create sample data and figure similar to yours df = pd.DataFrame({ 'order_date': ['2019-08-25 00:00:00','2019-08-30','2019-09-10','2019-09-10','2019-09-10','2019-09-15','2019-09-15','2020-02-01'], 'sales': [5,5,5,10,20,20,15,15], }) df['order_date'] = pd.to_datetime(df['order_date']) fig = px.line(df, x='order_date', y='sales', markers=True, ) fig.update_traces(marker_color='blue', line_color='darkgrey') ## between consecutive points: ## get the difference in time, and the average sales df_diff = pd.DataFrame({ 'time_diff': df['order_date'].diff(), 'sales': (df['sales'] + df['sales'].shift(-1)) / 2, }) df_diff['order_date'] = df['order_date'] + (df_diff['time_diff'] / 2).shift(-1) df_diff['time_diff'] = df_diff['time_diff'].shift(-1) y_axis_range = 1.25*(df.sales.max() - df.sales.min()) fig.add_trace( go.Scatter( x=df_diff["order_date"], y=df_diff["sales"] + 0.01*y_axis_range, text=df_diff["time_diff"].astype(str), mode='text', showlegend=False, ) ) fig.show()
Question: I need to read data from flask into react. React: const data = { name: 'John', email: 'john@example.com' }; axios.post('http://127.0.0.1:5000/api/data', data) .then(response => { console.log(response); }) .catch(error => { console.log(error); });  Flask: from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app, origins='http://localhost:3000') @app.route('/api/data', methods=['POST']) def process_data(): data = request.get_json() # обработка данных response_data = {'message': 'Sucsess'} return jsonify(response_data), 200 if __name__ == '__main__': app.run(debug=True)  And i take in browser an error: Access to XMLHttpRequest at 'http://127.0.0.1:5000/api/data' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. ShortPage.jsx:24 AxiosError {message: 'Network Error', name: 'AxiosError', code: 'ERR_NETWORK', config: {…}, request: XMLHttpRequest, …} How can i fix this?

Answer: The problem is that you are limiting the path /api/data to the http method POST @app.route('/api/data', methods=['POST']) CORS requires the client to make an OPTIONS 'pre-flight' call on that path to find out which origins are allowed (and other stuff like headers). Because your code only permits POST the OPTIONS call is declined and the CORS mechanism fails. The fix is to add the OPTIONS to the list of permitted methods on the route. @app.route('/api/data', methods=['POST', 'OPTIONS'])
Question: I wrote the code as follows to remove last character of Data Frames columns if it is number. Data Frames are stored as a values of dictionary in python and their name as a key of dictionary. my code doesn't produce the right result, how should I develop it? for key,val in df.items(): if key != "name1": for col in val.columns: m = re.search(r'\d+$', col) if m is not None: col = col[:-1] 

Answer: You can simplify your code by just replacing a single digit at the end of a column name with an empty string. This will not affect any values that don't end with a digit: for key, val in df.items(): if key != "name1": val.columns = val.columns.str.replace(r'\d$', '', regex=True)
Question: Pretty much anyone who works with IEEE floating-point values has run into NaN, or "not a number", at some point. Famously, NaN is not equal to itself. >>> x = float('nan') >>> x == x False  Now, I had come to terms with this, but there's a strange behavior I'm struggling to wrap my head around. Namely, >>> x in [x] True  I had always assumed that list.__contains__ was written something like def __contains__(self, element): for x in self: if element == x: return True return False  i.e., it used __eq__ on the relevant data type internally. And indeed it does. If I define a custom class with an __eq__ method of my own design, then I can verify that Python does in fact call __eq__ when doing the inclusion check. But then how can there exist a value x (NaN in our case) such that x == x is false but x in [x] is true? We can observe the same behavior with a custom __eq__ as well. class Example: def __eq__(self, other): return False x = Example() print(x == x) # False print(x in [x]) # True 

Answer: According to the docs it first uses the is operator to check for equality, and since x is x is True, x in [x] is also True:  For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression x in y is equivalent to any(x is e or x == e for e in y).  Note that identity (is) is different from equality (==). Also note that not all NaN values are represented by the same object, so if you try your test with two different NaN objects: >>> float('nan') in [float('nan')] False  you'll see different results.
Question: The official Enum HOWTO has this example: class Planet(Enum): MERCURY = (3.303e+23, 2.4397e6) VENUS = (4.869e+24, 6.0518e6) EARTH = (5.976e+24, 6.37814e6) MARS = (6.421e+23, 3.3972e6) JUPITER = (1.9e+27, 7.1492e7) SATURN = (5.688e+26, 6.0268e7) URANUS = (8.686e+25, 2.5559e7) NEPTUNE = (1.024e+26, 2.4746e7) def __init__(self, mass, radius): self.mass = mass # in kilograms self.radius = radius # in meters @property def surface_gravity(self): # universal gravitational constant (m3 kg-1 s-2) G = 6.67300E-11 return G * self.mass / (self.radius * self.radius)  >>> Planet.EARTH.value (5.976e+24, 6378140.0) >>> Planet.EARTH.surface_gravity 9.802652743337129  Suppose I'm doing something like this, and I want to treat the .values—the tuples like (3.303e+23, 2.4397e6)—as unstable implementation details of the Planet API. I don't want my API consumers to ever rely on them. Instead, I want them to use the properties that I explicitly expose myself, like .surface_gravity. Is there a conventional way to indicate this? I'm currently just adding a note in the docstring like this: class Planet(Enum): """.value is an implementation detail. Use .surface_gravity instead."""  But that seems too easy to miss. If it were a normal class, I would just make it ._value instead of .value. But here, .value is added automatically because I subclassed from Enum, and I don't see a way to override that.

Answer: The exact value of any enum is nearly always an implementation detail; the reason it's exposed at all is that sometimes it's useful to be able to access it. There are several ways to de-emphasize it's presence:  change the repr() of that enum class   def __repr__(self): return '<%s.%s>' % (self.__class__.__name__, self._name_)   have a custom __new__ to make the value obviously "wrong" to treat as surface gratity (or whatever) (probably combined with the __repr__ above):   def __new__(cls, mass, radius): member = object.__new__(cls): member._value_ = len(cls._member_names_) member.mass = mass member.radius = radius return member   you could combine with a dataclass (which automatically updates the repr):   @dataclass class PlanetData: mass: float radius: float class Planet(PlanetData, Enum): MERCURY = (3.303e+23, 2.4397e6) VENUS = (4.869e+24, 6.0518e6) # etc >>> Planet.VENUS <Planet.VENUS: mass=4.869e+24, radius=6051800.0>   Disclosure: I am the author of the Python stdlib Enum, the enum34 backport, and the Advanced Enumeration (aenum) library.
Question: I'm using MySQL (8.0.33-winx64, for Windows) with Python 3 and mysql.connector package. Initially my mysql-8.0.33-winx64\data folder was rather small: < 100 MB. Then after a few tests of CREATE TABLE..., INSERT... and DROP TABLE..., I notice that even after I totally drop the tables, the data folder keeps growing:  #innodb_redo folder seems to stay at max 100 MB #innodb_temp seems to be small binlog.000001: this one seems to be the culprit: it keeps growing even if I drop tables!  How to clean this data store after I drop tables, to avoid unused disk space with MySQL? Is it possible directly from Python3 mysql.connector API? Or from a SQL command to be execute (I already tried "OPTIMIZE" without success)? Or do I need to use an OS function manually (os.remove(...))?  Note: the config file seems to be in mysql-8.0.33-winx64\data\auto.cnf in the portable Windows version (non-used as a service, but started with mysqld --console) (no default config file is created after a first run of the server, we can create it in mysql-8.0.33-winx64\my.cnf)

Answer: You can disable the binary log, but only by setting disable_log_bin in your my.cnf file and restarting the MySQL Server. (See Disable MySQL binary logging with log_bin variable ) You can't change binary logging dynamically. See https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_log_bin You can make the binary log automatically expire old logs as it rolls over to a new binlog file. This helps to limit the overall storage. See https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds You do need to understand what the binary log is used for before you decide to disable it. You might need it! The binary log is commonly used for three or four things:  Replication  Point-in-time recovery  Change Data Capture (CDC) tools  A poor form of change auditing. A real audit log is better, but some sites don't have the audit log plugin installed.     can you edit to clarify the difference between undo log vs. bin log?  The binary log is for logging logical changes to your data. Nothing is written to the binary log until you COMMIT a transaction. It is not used for rollback, because by definition anything in the binary log has been committed. The binary log applies to all storage engines in MySQL. The undo log is only for the InnoDB storage engine. As you make changes to data during a transaction, the old version of the data is added to the undo log (this is also called the rollback segment in some documentation). So if you ROLLBACK, InnoDB can restore the original data. If you COMMIT, then the contents of the undo log for that transaction is discarded.  Notes:  setting disable_log_bin in my.cf and restarting the MySQL server won't delete old binlogs.  if you set disable_log_bin and restart the server first, and then do FLUSH LOGS; PURGE BINARY LOGS BEFORE NOW();, it won't delete old binlogs  you have to do FLUSH LOGS; PURGE BINARY LOGS BEFORE NOW(); first, and only then edit the config mysql-8.0.33-winx64\my.cnf to include: [mysqld] disable_log_bin  Then the old logs are deleted, and no new binlog will be created.
Question: Is it possible to use a containing object's attribute during the validation of a child object in a pydantic model? Given the json data: # example.json { "multiplier": 5, "field_1": { "value": 1 }, "field_2": { "value": 2 } }  and the corresponding Pydantic model: # example.py from pydantic import BaseModel, validator class Item(BaseModel): value: int class Container(BaseModel): multiplier: int field_1: Item field_2: Item  is it possible to use the Container object's multiplier attribute during validation of the Item values? For instance, I'd like to do something like this to Item at runtime: class Item(BaseModel): value: int @validator("value") @classmethod def validate_value(cls, value): return value # * multiplier # <--- can I get access to Container's multiplier here?  but I cannot determine if it possible to get access to the Container.multiplier value in a case like this? In my actual use case, the nesting is much, much deeper and so I would prefer not have the validator up at the Container level as access becomes fairly complicated, but I also do not want to duplicate the multiplier value down at the Item level? Is there any way to pass parameters up and down the object hierarchy within a model of this sort?

Answer: Simplest approach is to perform validation on the parent instead of the child: from pydantic import BaseModel, validator class Item(BaseModel): value: int class Container(BaseModel): multiplier: int field_1: Item field_2: Item @validator("field_1", "field_2") def validate_value(cls, v, values): """Validate each item""" m = values["multiplier"]  If you want to define the validator on the child, you can create a function and then call the validation function from the parent: class Item(BaseModel): value: int @classmethod def validate_value(cls, v, **kwargs): """Validate each item""" m = kwargs.get("multiplier") if m * v.value < 10: raise ValueError return v class Container(BaseModel): multiplier: int field_1: Item field_2: Item @validator("field_1", "field_2", pre=False) def validate_items(cls, v, values): return Item.validate_value(v, **values)  Another alternative is to pass the multiplier as a private model attribute to the children, then the children can use the pydantic validation function, but you'll still need to assign dynamically to the children: from pydantic import BaseModel, Field, validator class Item(BaseModel): multiplier: int # exclude from parent serialization, workaround for validation value: int @validator("value", pre=False) def validate_value(cls, v, values): """Validate each item""" m = values["multiplier"] if m * v < 10: raise ValueError return v class Container(BaseModel): multiplier: int field_1: Item = Field(..., exclude={'multiplier'}) field_2: Item = Field(..., exclude={'multiplier'}) @validator("field_1", "field_2", pre=True) def validate_items(cls, v, values): # Construct from a value, another workaround if isinstance(v, int): return Item(value=v, multiplier=values["multiplier"]) elif isinstance(v, Item): return Item(value=v.value, multiplier=values["multiplier"]) if __name__ == '__main__': c = Container( multiplier=1, field_1=11, field_2=22 )
Question: I have been trying to clean my data column by taking a part of the text out. Unfortunately cannot get my head around it. I tried using the .replace method in pandas series, but that did not seem to have worked df['Salary Estimate'].str.replace(' (Glassdoor est.)', '',regex=True) 0 $53K-$91K (Glassdoor est.) 1 $63K-$112K (Glassdoor est.) 2 $80K-$90K (Glassdoor est.) 3 $56K-$97K (Glassdoor est.) 4 $86K-$143K (Glassdoor est.) ... 922 -1 925 -1 928 $59K-$125K (Glassdoor est.) 945 $80K-$142K (Glassdoor est.) 948 $62K-$113K (Glassdoor est.) Name: Salary Estimate, Length: 600, dtype: object  What I expected was  0 $53K-$91K 1 $63K-$112K 2 $80K-$90K 3 $56K-$97K 4 $86K-$143K ... 922 -1 925 -1 928 $59K-$125K 945 $80K-$142K 948 $62K-$113K Name: Salary Estimate, Length: 600, dtype: object` 

Answer: If you enable regex, you have to escape regex symbol like (, ) or .: import re >>> df['Salary Estimate'].str.replace(re.escape(r' (Glassdoor est.)'), '',regex=True) 0 $53K-$91K 1 $63K-$112K 2 $80K-$90K 3 $56K-$97K 4 $86K-$143K Name: Salary Estimate, dtype: object # Or without import re module >>> df['Salary Estimate'].str.replace(r' \(Glassdoor est\.\)', '',regex=True) 0 $53K-$91K 1 $63K-$112K 2 $80K-$90K 3 $56K-$97K 4 $86K-$143K Name: Salary Estimate, dtype: object  You can also extract numbers: >>> df['Salary Estimate'].str.extract(r'\$(?P<min>\d+)K-\$(?P<max>\d+)K') min max 0 53 91 1 63 112 2 80 90 3 56 97 4 86 143
Question: I'm trying to count how many documents per project (among other things) have been registered, Registered also holds other values like DateTime and User (just a disclosure they are unrelated to my issue). Iterating through a list and querying ~54 times ends up being much faster than Annotate or Subquery (13 vs 27 seconds) I have 4 tables which go like this Project -> Batch -> Document -> Registered. Right now we have 54 Projects, 20478 Batches, 3231095 Documents and 312498 Registered. The timings listed below are for the entire view, not just the bits represented, but you can have an idea on their impact... projects = Project.objects.filter(completion_date__isnull=True) # "Fast", takes around 13.5 seconds to evaluate for project in list(projects.values('id')): count = Document.objects.filter(batch__project_id=project['id], registered__isnull=False) # Both methods below are much slower, taking around 27 seconds to evaluate # Simple annotation projects = projects.annotate(count=Count('batches__documents', filter=Q(batches__documents__registered__isnull=False), distinct=True)) # An attempt to simulate the first list iteration query without multiple db hits, ends up exactly as slow as annotate subq_registered = Document.objects.filter(batch__project_id=OuterRef('id'), registered__isnull=False).annotate(count=Func(F('id'), function='Count')).values('count') projects = projects.annotate(count=Subquery(subq_registered))  I expected Subquery to be faster than my first method, as it would mimic it, but its just as slow as simple count + annotation. Is there something I can do to speed any of my methods up? perhaps adding a Project foreign key to Document

Answer: I think you can use GROUP BY functionality here like this:  Document.objects.filter(batch__project__completion_date__isnull=True).values('batch__project').annotate(doc_count= Count('pk')).values('batch__project','doc_count')  More information on documentation Regarding the performance issue, there is no straight answer on why the performance is slow. Because (at least) the simple annotation makes single DB call, where others make more DB hits. There could be an issue with Database itself, maybe it has indexing issues (which needs debugging). I would recommend looking into this documentation regarding how to debug and optimize database related issues.
Question: Problem: I have a CSV file with a large amount of data, and I need to perform some data cleaning and filtering operations on it using Python. For instance, the CSV file contains a column with dates in the format "YYYY-MM-DD", but some of the entries have incorrect formatting or missing values. I need to clean up these entries so that they are all in the correct format and remove any rows that have missing dates. How can I clean and filter a large CSV file using Python with the shortest runtime possible? import csv # Read the CSV file into a list of dictionaries data = [] with open('data.csv') as csvfile: reader = csv.DictReader(csvfile) for row in reader: data.append(row) # Loop through the data and clean up the date column for i in range(len(data)): if 'date' in data[i]: date = data[i]['date'] if date: try: year, month, day = date.split('-') year = int(year) month = int(month) day = int(day) if year < 1000 or year > 9999 or month < 1 or month > 12 or day < 1 or day > 31: raise ValueError('Invalid date format') data[i]['date'] = f'{year}-{month:02d}-{day:02d}' except ValueError: del data[i]['date'] # Loop through the data and remove rows with missing dates clean_data = [] for row in data: if 'date' in row and row['date']: clean_data.append(row) 

Answer: I would recommend to use the pandas module which enables to handle csv data efficiently in Python. For instance, the following code solves your problem: import pandas as pd # Read the CSV file into a pandas dataframe df = pd.read_csv('data.csv') # Clean up the date column df['date'] = pd.to_datetime(df['date'], errors='coerce') # Remove rows with missing dates df.dropna(subset=['date'], inplace=True)  Please note that this does not anylonger check whether your date is in format "YYYY-MM-DD" but in any format which is suitable for dates. Since this is more flexible, it might be an advantage. Otherwise, you can simply modify the code to your needs.