eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am facing an issue while converting hex to decimal (learned from here) in pyspark. from pyspark.sql.functions import col, sha2, conv, substring # User data with ZIPs user_data = [ ("100052441000101", "21001"), ("100052441000102", "21002"), ("100052441000103", "21002"), ("user1", "21001"), ("user2", "21002") ] df_users = spark.createDataFrame(user_data, ["user_id", "ZIP"]) # Generate SHA-256 hash from the user_id df_users = df_users.withColumn("hash_key", sha2(col("user_id"), 256)) # Convert the hexadecimal hash (sha2 output) to decimal df_users = df_users.withColumn("hash_substr1", substring(col('hash_key'), 1, 16)) df_users = df_users.withColumn("hash_substr2", substring(col('hash_key'), 1, 15)) df_users = df_users.withColumn("hash_int1", conv(col('hash_substr1'), 16, 10).cast("bigint")) df_users = df_users.withColumn("hash_int2", conv(col('hash_substr2'), 16, 10).cast("bigint")) df_users.show()  The output I get is: +---------------+-----+--------------------+----------------+---------------+-------------------+-------------------+ | user_id| ZIP| hash_key| hash_substr1| hash_substr2| hash_int1| hash_int2| +---------------+-----+--------------------+----------------+---------------+-------------------+-------------------+ |100052441000101|21001|3cf4b90397964f6b2...|3cf4b90397964f6b|3cf4b90397964f6|4392338961672327019| 274521185104520438| |100052441000102|21002|e18aec7bb2a60b62d...|e18aec7bb2a60b62|e18aec7bb2a60b6| null|1015753888833888438| |100052441000103|21002|e55127f9f61bbe433...|e55127f9f61bbe43|e55127f9f61bbe4| null|1032752028895525860| | user1|21001|0a041b9462caa4a31...|0a041b9462caa4a3|0a041b9462caa4a| 721732164412679331| 45108260275792458| | user2|21002|6025d18fe48abd451...|6025d18fe48abd45|6025d18fe48abd4|6928174017724202309| 433010876107762644| +---------------+-----+--------------------+----------------+---------------+-------------------+-------------------+  Note that hash_int1 is null for 2nd and 3rd records. However, when I try to get the corresponding int using python, I get some value: hexes = ["e18aec7bb2a60b62", "e18aec7bb2a60b6", "e55127f9f61bbe43", "e55127f9f61bbe4"] [int(h, 16) for h in hexes] [16252062221342215010, 1015753888833888438, 16524032462328413763, 1032752028895525860]  The values are same when they are not null. The final objective is to generate replicable random values df_users = df_users.withColumn("random_value", (col("hash_int1") % 10**12) / 10**12) 

Answer: Spark's LongType range is -9223372036854775808 to 9223372036854775807. However, your value 16524032462328413763 is outside of this range so it cannot store as LongType. If you remove .cast("bigint"), you can see that your values won't be null and have correct values.
Question: I’m working on a Python project where my goal is to generate barcodes in the data:image/png;base64 format, without any human-readable footer text. Additionally, I need to adjust the size (height and width) and DPI (dots per inch) of the barcode, but I'm encountering some difficulties. Specifically, I have two issues: I cannot remove the human-readable footer text that appears below the barcode. I am unable to customize the size (height, width) and DPI of the generated barcode. I am using the python-barcode library along with the ImageWriter to generate the barcode image. I have tried using options like text=None to remove the text and various writer_options to control the size and resolution, but nothing seems to work as expected. Here’s what I have tried so far: import barcode from barcode.writer import ImageWriter from io import BytesIO import base64 def generate_barcode_without_text(serial_no): barcode_instance = barcode.Code128(serial_no, writer=ImageWriter()) writer_options = { 'text': None, # Disable text under the barcode 'module_width': 0.2, # Adjust the width of the barcode 'module_height': 15, # Adjust the height of the barcode 'dpi': 300, # Set the DPI for the image 'quiet_zone': 6 # Set the quiet zone around the barcode } image_stream = BytesIO() barcode_instance.write(image_stream, writer_options=writer_options) image_stream.seek(0) barcode_base64 = base64.b64encode(image_stream.read()).decode('utf-8') return barcode_base64 

Answer: I think the docs are slightly off for the current latest version 0.15-1. I was able to change the size and remove the human readable barcode text from under the image with the following code: def generate_barcode_without_text(serial_no): barcode_instance = barcode.Code128(serial_no, writer=ImageWriter()) writer_options = { 'write_text': False, # Disable text under the barcode 'module_width': 0.8, # Adjust the width of the barcode 'module_height': 25, # Adjust the height of the barcode 'dpi': 300, # Set the DPI for the image 'quiet_zone': 6 # Set the quiet zone around the barcode } image_stream = BytesIO() barcode_instance.write(image_stream, options=writer_options) image_stream.seek(0) barcode_base64 = base64.b64encode(image_stream.read()).decode('utf-8') return barcode_base64  The option write_text seems to control whether the text under the barcode is written or not. The current docs don't mention this option and say setting the font_size to 0 suppresses the text, but this yields an error for me. The parameter name for passing the writer options to the barcode class is options instead of writer_options too. Here is the resulting barcode:
Question: I'm following a tutorial on youtube on Fourier Optics in python, to simulate diffraction of light through a slit.  The video in question  Source Code of video  Now, I'm trying to implement the get_U(z, k) function and then display the corresponding plot below it, as shown in the video (I've got barebones knowledge about this topic), however, i just can't seem to get the plot working (white plot is visible the entire time). Upon inspection, I've discovered that the U variable just consists of a bunch of (nan+nanj) values, which I think shouldn't be the case. I've crosschecked the formula and it looks perfect. I also realise that, sometimes the np.sqrt() has to deal with negative values, but adding neither a np.abs() nor a np.where()(to convert negatives to zero) gives me the intended output.  My code: import numpy as np import scipy as sp from scipy.fft import fft2, ifft2, fftfreq, fftshift import matplotlib.pyplot as plt import pint plt.style.use(['grayscale']) u = pint.UnitRegistry() D = 0.1 * u.mm lam = 660 * u.mm x = np.linspace(-2, 2, 1600) * u.mm xv, yv = np.meshgrid(x, x) U0 = (np.abs(xv) < D/2) * (np.abs(yv) < 0.5 * u.mm) U0 = U0.astype(float) A = fft2(U0) kx = fftfreq(len(x), np.diff(x)[0]) * 2 * np.pi kxv, kyv = np.meshgrid(kx, kx) def get_U(z, k): return ifft2(A*np.exp(1j*z.magnitude*np.sqrt(k.magnitude**2 - kxv.magnitude**2 - kyv.magnitude**2))) k = 2*np.pi/(lam) d = 3 * u.cm U = get_U(d, k) plt.figure(figsize=(5, 5)) plt.pcolormesh(xv, yv, np.abs(U), cmap='inferno') plt.xlabel('$x$ [mm]') plt.ylabel('$y$ [mm]') plt.title('Single slit diffraction') plt.show() 

Answer: Your units of lam are wrong - if you intend to use pint (but I suggest that you don't) then they should be in nm, not mm. When you have made that change I suggest that you remove all reference to pint and mixed units and work entirely in a single set of length units (here, m). This is because units appear to be stripped when creating some of the numpy arrays. You can use scientific notation (e.g. 1e-9) to imply the units. Then you get what I think you require.  import numpy as np import scipy as sp from scipy.fft import fft2, ifft2, fftfreq, fftshift import matplotlib.pyplot as plt plt.style.use(['grayscale']) D = 0.1 * 1e-3 lam = 660 * 1e-9 x = np.linspace(-2, 2, 1600) * 1e-3 xv, yv = np.meshgrid(x, x) U0 = (np.abs(xv) < D/2) * (np.abs(yv) < 0.5 * 1e-3) U0 = U0.astype(float) A = fft2(U0) kx = fftfreq(len(x), np.diff(x)[0]) * 2 * np.pi kxv, kyv = np.meshgrid(kx, kx) def get_U(z, k): return ifft2(A*np.exp(1j*z*np.sqrt(k**2 - kxv**2 - kyv**2))) k = 2*np.pi/(lam) d = 3 * 1e-2 U = get_U(d, k) plt.figure(figsize=(5, 5)) plt.pcolormesh(xv*1e3, yv*1e3, np.abs(U), cmap='inferno') plt.xlabel('$x$ [mm]') plt.ylabel('$y$ [mm]') plt.title('Single slit diffraction') plt.show()
Question: I use the MNIST dataset that contains 28x28 grayscale images represented as numpy arrays with 0-255 values. I'd like to convert images to black and white only (0 and 1) so that pixels with a value over 128 will get the value 1 and pixels with a value under 128 will get the value 0. Is there a simple method to do so?

Answer: Yes. Use (arr > 128) to get a boolean mask array of the same shape as your image, then .astype(int) to cast the bools to ints: >>> import numpy as np >>> arr = np.random.randint(0, 255, (5, 5)) >>> arr array([[153, 167, 141, 79, 58], [184, 107, 152, 215, 69], [221, 90, 172, 147, 125], [ 93, 35, 125, 186, 187], [ 19, 72, 28, 94, 132]]) >>> (arr > 128).astype(int) array([[1, 1, 1, 0, 0], [1, 0, 1, 1, 0], [1, 0, 1, 1, 0], [0, 0, 0, 1, 1], [0, 0, 0, 0, 1]])
Question: I am managing a project with uv. My project includes - src - app.py - constants.py - notebooks - testing.ipynb - pyproject.toml  where my pyproject.toml is [project] name = "benchmark-extractor" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.13" dependencies = [ "anthropic>=0.42.0", "ipykernel>=6.29.5", "markitdown>=0.0.1a3", "pandera>=0.22.1", "tabula-py>=2.10.0", ]  I want to import app.py and constants.py into notebooks/testing.ipynb, for experimentation purposes. When I do this naively, e.g. import src.app, I get a ModuleNotFoundError. I believe I could approach this using uv pip install -e . (e.g. Sibling package imports) but when I try that, I get ERROR: Package 'benchmark-extractor' requires a different Python: 3.11.3 not in '>=3.13'.

Answer: Add this section into pyproject.toml: [tool.uv] package = true  Then run uv sync Now you should be able to import app in notebooks/testing https://docs.astral.sh/uv/concepts/projects/config/#project-packaging
Question: I'm having difficulty with Pandas when trying to convert this column to a date. The table doesn't include a year, so I think that's making the conversion difficult.  28 JUL Unnamed: 0 Alura *Alura - 7/12 68,00 0 28 JUL NaN Passei Direto S/A. - 3/12 19,90 1 31 JUL NaN Drogarias Pacheco 25,99 2 31 JUL NaN Mundo Verde - Rj - Sho 5,90 3 31 JUL NaN Paypal *99app 4,25 4 04 AGO NaN Saldo em atraso 1.091,17 5 04 AGO NaN Crédito de atraso 1.091,17 6 06 AGO NaN Apple.Com/Bill 34,90 7 07 AGO NaN Pagamento em 07 AGO 1.091,17 8 07 AGO NaN Juros de atraso 16,86 9 07 AGO NaN IOF de atraso 4,43 10 07 AGO NaN Multa de atraso 21,91 11 08 AGO NaN Apple.Com/Bill 21,90 12 09 AGO NaN Google Youtubepremium 20,90 13 10 AGO NaN Amazon.Com.Br 41,32 14 12 AGO NaN Uber *Uber *Trip 17,91 15 12 AGO NaN Uber *Uber *Trip 16,94 16 12 AGO NaN Mia Cookies 47,50 17 13 AGO NaN Uber *Uber *Trip 16,96 18 13 AGO NaN Uber *Uber *Trip 19,98 19 16 AGO NaN Uber *Uber *Trip 11,93 20 16 AGO NaN Uber *Uber *Trip 9,97 21 18 AGO NaN Uber *Uber *Trip 9,91 22 22 AGO NaN Uber *Uber *Trip 9,96 23 23 AGO NaN Amazonprimebr 14,90 24 27 AGO NaN Paypal *Sacocheiotv 15,00 25 27 AGO NaN Pag*Easymarketpleno 6,50  I tried to transform it using this code, but it's not working: df["Data"] = pd.to_datetime(df["Data"], format="%d %b", errors="coerce")  Incorrect output: Data Local Valor 0 1900-07-28 Alura *Alura - 7/12 68,00 1 1900-07-28 Passei Direto S/A. - 3/12 19,90 2 1900-07-31 Drogarias Pacheco 25,99 3 1900-07-31 Mundo Verde - Rj - Sho 5,90 4 1900-07-31 Paypal *99app 4,25 7 NaT Apple.Com/Bill 34,90 9 NaT Juros de atraso 16,86 10 NaT IOF de atraso 4,43 11 NaT Multa de atraso 21,91 12 NaT Apple.Com/Bill 21,90 13 NaT Google Youtubepremium 20,90 14 NaT Amazon.Com.Br 41,32 15 NaT Uber *Uber *Trip 17,91 16 NaT Uber *Uber *Trip 16,94 17 NaT Mia Cookies 47,50 18 NaT Uber *Uber *Trip 16,96 19 NaT Uber *Uber *Trip 19,98 20 NaT Uber *Uber *Trip 11,93 21 NaT Uber *Uber *Trip 9,97 22 NaT Uber *Uber *Trip 9,91 23 NaT Uber *Uber *Trip 9,96 24 NaT Amazonprimebr 14,90 25 NaT Paypal *Sacocheiotv 15,00 26 NaT Pag*Easymarketpleno 6,50  Could someone help me with this?

Answer: This looks like Brazilian Portuguese, you should install the pt_BR locale on your machine, then run: import locale locale.setlocale(locale.LC_ALL, 'pt_BR.UTF-8') df['Data_converted'] = pd.to_datetime(df['Data'], format='%d %b', errors='coerce')  Output:  Data Data_converted 0 28 JUL 1900-07-28 1 04 AGO 1900-08-04  And, if you want to force the year: df['Data_converted'] = pd.to_datetime('2025 ' + df['Data'], format='%Y %d %b', errors='coerce')  Output:  Data Data_converted 0 28 JUL 2025-07-28 1 04 AGO 2025-08-04
Question: I am working on a python package chemcoord with several subpackages, some of whom should be exposed to the root namespace. The repository is here, the relevant __init__.py file is here. For example there is a chemcoord.cartesian_coordinates.xyz_functions that should be accessible as chemcoord.xyz_functions Accessible, in particular, means that the user should be able to write: from chemcoord.xyz_functions import allclose  If I write in my __init__.py import chemcoord.cartesian_coordinates.xyz_functions as xyz_functions  then I can use chemcoord.xyz_functions in the code, but I cannot do from chemcoord.xyz_functions import allclose  If I do the additional ugly/hacky (?) trick of modifying sys.modules in the __init__.py as in import sys sys.modules["chemcoord.xyz_functions"] = xyz_functions  then I can write from chemcoord.xyz_functions import allclose  But it feels ugly and hacky. Recently I got warnings from PyLance about  Import "chemcoord.xyz_functions" could not be resolved  Which leads to my two questions:  Is my approach of reexposing the submodule correct, or is there a cleaner way? If the answer to question 1 is solved and I still get warnings from PyLance, is there a bug in PyLance? 

Answer: nit: # -*- coding: utf-8 -*- hasn't been needed in python source files for a very very long time, given that it is default starting with interpreter 3.0. And 3.8 went EOL last year, so really you only need to worry about 3.9 and later. And these seem odd: ... import Cartesian as Cartesian, ... import Zmat as Zmat sys.modules I completely agree with you that the sys.modules["chemcoord.xyz_functions"] assignment is not appropriate. And I agree that apps consuming the Public API should be able to use a short from chemcoord.xyz_functions import allclose, dot. Organizing everything under src/ is very nice and I thank you for that. The entire cartesian_coordinates/ folder keeps things tidy for the package developer and makes good sense. However, you might possibly wish to "hide" a code module by renaming it to _xyz_functions.py, with leading _ underscore. (I don't know what promises you've made to app developers in v2.1.2 and previous, which might require you to still expose some or all of that.) new module Focusing on the OP question, you complain that an app author currently cannot do from chemcoord.xyz_functions import allclose  Honestly, it's very simple. You're just a little hung up on that src/chemcoord/cartesian_coordinates/xyz_functions.py filename. You're thinking that is the "right" location, but it's just a private implementation detail, separate from the Public API you choose to expose, which plays into why we might want to hide it. I claim that what you want to do, to answer the original question, is instead of editing an __init__.py module, you want to create a new src/chemcoord/xyz_functions.py module. It can pull in allclose(), dot(), etc., and make them visible with conveniently short names for app authors.   Yes, there's a cleaner way, just define a new public module. No, there's no trouble with pylance here. (Also, you might prefer to run $ pyright . .)
Question: quick problem description: pyrtools imshow function giving me different and negative ranges details: im following the tutorial at https://pyrtools.readthedocs.io/en/latest/tutorials/01_tools.html since i dont have .pgm image, i'm using the below .jpg image.  here is the modified python code import matplotlib.pyplot as plt import pyrtools as pt import numpy as np import cv2 # Load the JPG image oim = cv2.imread('/content/ein.jpg') print(f"input image shape: {oim.shape}") # Convert to grayscale if it's an RGB image if len(oim.shape) == 3: # Check if the image has 3 channels (RGB) oim_gray = cv2.cvtColor(oim, cv2.COLOR_BGR2GRAY) else: oim_gray = oim # Already grayscale print(f"grayscale image shape: {oim_gray.shape}") # Check the range of the oim_gray print(f"value range of oim_gray: {np.min(oim_gray), np.max(oim_gray)}") # Subsampling imSubSample = 1 im = pt.blurDn(oim_gray, n_levels=imSubSample, filt='qmf9') # Check the range of the subsampled image print(f"value range of im: {np.min(im), np.max(im)}") # Display the original and subsampled images pt.imshow([oim_gray, im], title=['original (grayscale)', 'subsampled'], vrange='auto2', col_wrap=2);  and here is the output input image shape: (256, 256, 3) grayscale image shape: (256, 256) value range of oim_gray: (4, 245) value range of im: (5.43152380173187, 251.90158197570906)  with the given image  as you can see, from the printouts, both the images oim_gray and im are in positive range. there aren't any negative values on neither im and oim_gray. but when checking the output image, i see the range -38 & 170 (please check the text over the output image). this doesn't make any sense, and don't understand. can you help on this to understand?

Answer: That is behavior of pyrtools, specifically decided by vrange='auto2'.  'auto2': all images have same vmin/vmax, which are the mean (across all images) minus/plus 2 std dev (across all images)  (documentation in source) If you don't like that library's behavior, you can file a bug at https://github.com/LabForComputationalVision/pyrtools/issues
Question: In Python-Polars, it is easy to calculate the Sum of all the lists in an array with polars.Expr.list.sum. See the example below for the sum: df = pl.DataFrame({"values": [[[1]], [[2, 3], [5,6]]]}) df.with_columns( sum=pl.concat_list(pl.col("values")).list.eval( pl.element().list.sum())) shape: (2, 2) ┌──────────────────┬───────────┐ │ values ┆ sum │ │ --- ┆ --- │ │ list[list[i64]] ┆ list[i64] │ ╞══════════════════╪═══════════╡ │ [[1]] ┆ [1] │ │ [[2, 3], [5, 6]] ┆ [5, 11] │ └──────────────────┴───────────┘  I am trying to define the same logic for the product and the division. Since it is not available in the current version of Polars (1.19). To do this, I am using pl.reduce, but it does not seem to work as expected: df.with_columns( sum=pl.concat_list(pl.col("values")).list.eval( pl.reduce(lambda e1, e2: e1*e2,pl.element()))) shape: (2, 2) ┌──────────────────┬──────────────────┐ │ values ┆ sum │ │ --- ┆ --- │ │ list[list[i64]] ┆ list[list[i64]] │ ╞══════════════════╪══════════════════╡ │ [[1]] ┆ [[1]] │ │ [[2, 3], [5, 6]] ┆ [[2, 3], [5, 6]] │ └──────────────────┴──────────────────┘  Would you have any suggestion on how to implement the above using a single expression context?

Answer:  pl.Expr.list.eval() to get into list context. pl.element() to get access to element within list context. pl.Expr.product() to calculate product. pl.Expr.list.first() to get the result as scalar.  df.with_columns( product = pl.col.values.list.eval( pl.element().list.eval( pl.element().product() ).list.first() ) )  shape: (2, 2) ┌──────────────────┬───────────┐ │ values ┆ product │ │ --- ┆ --- │ │ list[list[i64]] ┆ list[i64] │ ╞══════════════════╪═══════════╡ │ [[1]] ┆ [1] │ │ [[2, 3], [5, 6]] ┆ [6, 30] │ └──────────────────┴───────────┘
Question: I'm trying to scrape the number of followers of an array of username. I'm using BeautifulSoup. The code I'm using is the following import requests from bs4 import BeautifulSoup def instagram_followers(username): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3' } response = requests.get(f'https://www.instagram.com/{username}/') soup = BeautifulSoup(response.text, 'html.parser') info = soup.find('meta', property='og:description') if info: followers = info['content'].split(" ")[0] return followers else: return -1  The function always returns -1

Answer: Code works fine in focus of your question, so issue is not reproducable, without any additional information. Check following:  response.status_code as first indicator, may you scrape to aggressivly and the server will handle this by blocking your ip. also implement your headers, they are not used in your code  import requests from bs4 import BeautifulSoup def instagram_followers(username): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3' } response = requests.get(f'https://www.instagram.com/{username}/', headers=headers) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') info = soup.find('meta', property='og:description') if info: followers = info['content'].split(" ")[0] return followers else: return -1 else: print('Something went wrong with your request: ' + response.status_code) instagram_followers('thestackoverflow')  Output: 52K
Question: I am trying to iterate multiple column rows and multiply nth row to n+1 row after that add columns. I tried below code and it's working fine. Is there any other simply way to achieve the subtraction and multiplication part together? import pandas as pd df = pd.DataFrame({'C': ["Spark","PySpark","Python","pandas","Java"], 'F' : [2,4,3,5,4], 'D':[3,4,6,5,5]}) df1 = pd.DataFrame({'C': ["Spark","PySpark","Python","pandas","Java"], 'F': [1,2,1,2,1], 'D':[1,2,2,2,1]}) df = pd.merge(df, df1, on="C") df['F_x-F_y'] = df['F_x'] - df['F_y'] df['D_x-D_y'] = df['D_x'] - df['D_y'] for index, row in df.iterrows(): df['F_mul'] = df['F_x-F_y'].mul(df['F_x-F_y'].shift()) df['D_mul'] = df['D_x-D_y'].mul(df['D_x-D_y'].shift()) df['F+D'] = df['F_mul'] + df['D_mul']  Output -  C F_x D_x F_y D_y F_x-F_y D_x-D_y F_mul D_mul F+D 0 Spark 2 3 1 1 1 2 NaN NaN NaN 1 PySpark 4 4 2 2 2 2 2.0 4.0 6.0 2 Python 3 6 1 2 2 4 4.0 8.0 12.0 3 pandas 5 5 2 2 3 3 6.0 12.0 18.0 4 Java 4 5 1 1 3 4 9.0 12.0 21.0 

Answer: First remove iterating by iterrows, then is possible simplify a generalize solution by: cols = ['F','D'] for col in cols: s = df[f'{col}_x'].sub(df[f'{col}_y']) df[f'{col}_mul'] = s.mul(s.shift()) df['+'.join(cols)] = df.filter(like='mul').sum(axis=1, min_count=1) print (df) C F_x D_x F_y D_y F_mul D_mul F+D 0 Spark 2 3 1 1 NaN NaN NaN 1 PySpark 4 4 2 2 2.0 4.0 6.0 2 Python 3 6 1 2 4.0 8.0 12.0 3 pandas 5 5 2 2 6.0 12.0 18.0 4 Java 4 5 1 1 9.0 12.0 21.0  Another idea is processing all columns together - advantage is dont need specify columns for processing: df1 = (df.filter(like='x').rename(columns=lambda x: x.replace('x','mul')) .sub(df.filter(like='y').rename(columns=lambda x: x.replace('y','mul')))) df2 = df1.mul(df1.shift()) df = df.join(df2) df['+'.join(x.replace('_mul','') for x in df2.columns)] = df2.sum(axis=1, min_count=1) print (df) C F_x D_x F_y D_y F_mul D_mul F+D 0 Spark 2 3 1 1 NaN NaN NaN 1 PySpark 4 4 2 2 2.0 4.0 6.0 2 Python 3 6 1 2 4.0 8.0 12.0 3 pandas 5 5 2 2 6.0 12.0 18.0 4 Java 4 5 1 1 9.0 12.0 21.0