Automating Forms Submissions with Python | by Lucas Soares | Towards Data Science

Filling out online forms is annoying. However, so many people that work desk jobs sometimes still have to spend quite a bit of time filling out online forms for work. So, why not use Python to automate that process and save some valuable time?
In this article I will show you how to use Python to automate the process of filling out submission forms.
If you prefer video check out my Youtube video here:
For this example I will create a simple google forms survey, which I will then inspect to gather the necessary web elements needed to perform my automation.
The overall steps will be:
Set up your answers as a .csv fileInspect the necessary web elements by type of inputSet up the url and the browserAccess the web addressSend the keys for text based inputsSend clicks for checkbox type inputsAccess and click the submit button to finishLooping over the database and submitting the answers for each user
Set up your answers as a .csv file
Inspect the necessary web elements by type of input
Set up the url and the browser
Access the web address
Send the keys for text based inputs
Send clicks for checkbox type inputs
Access and click the submit button to finish
Looping over the database and submitting the answers for each user
For simplicity purposes, the form I created on google forms is made of three simple questions:
What is your name?What is your age?What is your favorite color?
What is your name?
What is your age?
What is your favorite color?
I know I know...very original, but this is just for demonstration purposes so bare with me. The actual form looks like this:
Now, let’s set up a .csv file to store our answers. I will simulate a simple database of names, ages and color choices using the faker library, the numpy.rando.randint() method and pandas for the final dataframe.
from selenium import webdriverimport pandas as pdimport timeimport jsonfrom datetime import datetimeimport pathlibimport globimport syssys.path.append(".")import fakerimport numpy as npimport pandas as pdf = faker.Faker()colors = ["Blue","Pink","Black","White","Green"]names = [f.name() for _ in range(100)]ages =  [np.random.randint(18,65) for _ in range(100)]color_choices = [np.random.choice(colors,1)[0] for _ in range(100)]database = pd.DataFrame(dict(names=names, ages=ages, colors = color_choices))database.to_csv("submission_form_database.csv", index=False)database.head()
Perfect! Now that we have our database, let’s set up the browser automation.
Now, I just go to the preview link of my google forms, or the actual web address of the survey for real case scenarios, and collect the names of the classes for the different questions in terms of input type.
For this case, I have three questions where 2 are text based and one is multi-select, so I inspect each element in the questions by pressing CTRL+Shift+I if you are in chrome, then going to: copy -> copy element where I copy the elements for each question which should look like this:
Web elements for question 1 and 2: <input type="text" class="quantumWizTextinputPaperinputInput exportInput" jsname="YPqjbf" autocomplete="off" tabindex="0" aria-labelledby="i1" aria-describedby="i2 i3" required="" dir="auto" data-initial-dir="auto" data-initial-value="">
Web element for question 3 (multi-select) <div class="appsMaterialWizToggleRadiogroupOffRadio exportOuterCircle"><div class="appsMaterialWizToggleRadiogroupOnRadio exportInnerCircle"></div></div>
Now, I just copy the class names for each type and set it to two different variables along with the xpath for the submit button:
text_question_element_class = "quantumWizTextinputPaperinputInput"checkbox_question_element_class = "appsMaterialWizToggleRadiogroupOffRadio"submit_element_class = '//*[@id="mG61Hd"]/div[2]/div/div[3]/div[1]/div/div/span/span'
Perfect, now we are ready for the automation part!
url = "https://forms.gle/WY7E9N8wkiMtziTD9"driver = webdriver.Chrome(executable_path="./chromedriver")
Here, we just set the variables for the url of the submission form, and the browser, which in my case is Google Chrome. To use it you will need to download the chromedriver that you can find here.
driver.get(url)
Now, we write a function that takes care of answering the text based questions by using the element class provided. This function will take the dataframe as input along with the class name, the user ID, and the driver controlling the browser and will submit the appropriate answer.
def answerNameAge(driver, df, element_class, user_id):    name = df["names"][user_id]    age = df["ages"][user_id]    text_answers = [name, str(age)] # following the order in the form    text_questions = driver.find_elements_by_class_name(element_class)    for a,q in zip(text_answers,text_questions):        q.send_keys(a)        return driver
Now, we set up the function to fill in the check box questions, for this one we will need a dictionary that matches the colors with their corresponding order in the form.
color_index_dict = {"Blue": 0, "Pink": 1, "Black": 2, "White": 3, "Green": 4}def answerCheckBox(driver, df, element_class, user_id):    color_answer = df["colors"][user_id]    color_answer_index = color_index_dict[color_answer]    driver.find_elements_by_class_name(element_class)[color_answer_index].click()            return driver
Great, now all we have to do is submit the form!
Here, we just write a simple function that accesses the xpath for the submission button obtained before and clicks it.
def submit(driver, element_class):    driver.find_element_by_xpath(element_class).click()    return driver
Done! Now, we loop over the database and submit the answers for each user.
Let’s just create a simple loop over our dataframe containing the data and submit the answers one by one.
df = pd.read_csv("./submission_form_database.csv")text_question_element_class = "quantumWizTextinputPaperinputInput"checkbox_question_element_class = "appsMaterialWizToggleRadiogroupOffRadio"url = "https://forms.gle/WY7E9N8wkiMtziTD9"driver = webdriver.Chrome(executable_path="./chromedriver")for user_id in range(len(df)):    driver.get(url)        driver.maximize_window()    driver = answerNameAge(driver, df, text_question_element_class, user_id)    driver = answerCheckBox(driver, df, checkbox_question_element_class, user_id)    driver = submit(driver, submit_element_class)
There we have it, we automated the boring process of submitting multiple forms! In my google forms response stats page I see this now:
This can be useful for individuals that often find themselves filling out online forms with similar structure, as well as people that work desk jobs and have to perform these kinds of submissions for clients or people in their organization.
Whatever repeatable task we do online should be automated so we can focus only on what matters. One helpful thing to do when trying to detect automatable tasks is to ask: do I do this periodically? Does it involve repeatable processes with little or no smart decision making involved? If the answer for both questions are yes then you should automate that task.
In this article you learned:
How to automate the process of filling out submissions forms like google forms using Python’s selenium framework
Automating submission forms involves: setting up a dataframe with the data for the forms, inspecting the web elements from the desired web address to know how to navigate the online form, writing a function to answer questions based on input type, writing a simple function to click on relevant buttons like the submit button
If you liked this post connect with me on Twitter, LinkedIn, Instagram, and follow me on Medium. Thanks and see you next time! :)