fabiochiu/medium-articles huggingface dataset:
Uniqgo — A Go service that tells me when Uniqlo clothes are in stock
By: Matthew Kim
Tags: Selenium, Docker, Golang

How is Uniqgo Implemented?

The interesting thing about Go programs is that they are organized into packages, defined as “collections of source files in the same directory that are compiled together”. And similar to how the Java main method is the entry point of Java programs, Go has a main function found in package main that serves as the entry point for Go programs. More info on how Go code is organized can be found here.

Theoretically, I could throw all my code inside package main . But after taking CIS 350 (Software Design & Engineering) last semester, I learned that putting all your code in one file is a big no-no. So I’ve decided to organize Uniqgo into four packages, briefly outlined below.

Service : Handles connecting to the Selenium server and starting/stopping the Uniqgo service

: Handles connecting to the Selenium server and starting/stopping the Uniqgo service Uniqlo : Creates and runs the cron job to check when the “Gudelicous” collection is in stock

: Creates and runs the cron job to check when the “Gudelicous” collection is in stock Notifier : Contains a function to sends me an email

: Contains a function to sends me an email Main: Calls the function to start the Uniqgo service

Let’s walk through the code in each package.

Service

service.go contains two functions, Start() and Stop() :

What Start() does up to line 21 is pretty straightforward — I’m loading in a config file and getting a map of my email login info (using this package), connecting to the Selenium server and creating a WebDriver client (using this package), and calling the function in the Uniqlo package that will start up the cron job, which takes in our config map and WebDriver client as arguments.

To allow Uniqgo to shut down gracefully, I’m going to be using some of Go’s awesome concurrency tools (check out this article about concurrency in Go). Inside a Goroutine, I’m going to be listening for the SIGTERM signal on a channel — once that signal is received by the channel, Uniqgo will call Stop() , which stops the cron job. And by using a WaitGroup, I can ensure that Start() will exit only when the SIGTERM signal is received.

Uniqlo

Let’s take a look at uniqlo.go :

Spawn() handles creating and starting the cron job. In order to have some control on when to stop the cron job from running, we’re going to use a select statement inside a Goroutine — this will block forever until stopCh is closed, which occurs when the SIGTERM signal is received. If you take a look back at service.go , when the SIGTERM signal is received, Stop() in service.go is called, which calls Stop() in uniqlo.go and ultimately closes stopCh .
Using Python and Selenium to get coordinates from street addresses
By: Khalid El Mouloudi
Tags: Mapping, Selenium, Web Scraping, Pandas, Python

Exploring Google Maps :

rawpixel.com

Before any web scraping job, it is essential to explore the website you want to extract data from. In our case, it’s Google Maps.

First, let’s study how searching for a full address using the search bar inside Google Maps affects the URL of the result page. For this, I’ll go with the fictitious address Grinch house mount crumpit whoville because I want Google Maps to return no results :

As you can see above, we get www.google.com/maps/search/ followed by the address we searched for. In other words, if we want to search for an address XYZ within Google Maps, all we have to do is use the URL www.google.com/maps/search/XYZ , without having to interact with the search bar itself.

The idea here is to generate a new column within mel where we combine www.google.com/maps/search/ with every Full_Address we have in our dataframe mel, and then have Selenium iterate over them visiting the URLs one after the other.

Let’s create that new Url column :

mel['Url'] = ['https://www.google.com/maps/search/' + i for i in mel['Full_Address'] ]

Now that we have a column containing all the URLs we want to crawl, let’s take a look at the address G Se 11 431 St Kilda Rd Melbourne for example :

www.google.com/maps/search/G Se 11 431 St Kilda Rd Melbourne

The above link gives us :

The above address corresponds to the charity Australian Nurses Memorial Centre. Let’s search for it on Google Maps by name :

We get the exact same spot, but not the same coordinates in the URL. That’s because the coordinates in the URL are linked to how the map is centered and not to the marker (they change if you zoom in or out). That’s why we’re going to extract the coordinates directly from the source code of the page itself.

To view the source code, right click on a blank space within the page (outside of the map) and choose View Page Source (CTRL+U or Command+U in Mac). Now search for -37.8 or 144.9 within the source page :

You’ll find the coordinates we are seeking in many places throughout the hot mess that is the source code. But they are mostly useful to us if they are comprised within an HTML tag we can target. Luckily, there is one meta tag we can make useful here :

For now, let’s note that it’s a meta tag with an attribute content containing the URL we want to extract, and an attribute itemprop with the value image which can be used to identify and target this particular meta tag.
How to setup Selenium on a Linux VM
By: Zhun Hung
Tags: Selenium, Linux, Automation, Web Scraping, Python

How to setup Selenium on a Linux VM

Photo by Nicolas Picard on Unsplash

Selenium is a popular framework for testing web applications. You’re probably here because you have written and tested your web scraping/testing script on your local machine. Now, you’re wondering…

“How can I automatically run my Selenium script every X hours/day/week?”

Therefore, this guide will show you how you can set up a Linux Virtual Machine (VM) with Selenium and Chromedriver to do exactly that. Although you can always use a Windows OS but let’s focus on a Linux OS for this guide.

Installation

To get selenium and Chromedriver running on your local machine, it can be broken down into 3 simple steps:

Install dependencies Install Chrome binary and Chromedriver Install Selenium

Step 1

Whenever you get a new Linux machine, always update the packages first. Then install the necessary dependencies.

Step 2

In order for Chromedriver to work on Linux, you’ll have to install Chrome binary.

For Chromedriver version, it’ll be dependent on your Chrome binary version.

Step 3

Last but not least, install Selenium.

And… you’re all set!

Testing your script

All that is left to do is to check if Selenium and Chromedriver are installed correctly and you’re able to run a python script that uses Selenium.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options CHROMEDRIVER_PATH = '/usr/local/bin/chromedriver'

WINDOW_SIZE = "1920,1080" chrome_options = Options()

chrome_options.add_argument("--headless")

chrome_options.add_argument("--window-size=%s" % WINDOW_SIZE)

chrome_options.add_argument('--no-sandbox')

chrome_options=chrome_options

)

driver.get("

print(driver.title)

driver.close() driver = webdriver.Chrome(executable_path=CHROMEDRIVER_PATH,chrome_options=chrome_optionsdriver.get(" https://www.google.com ")print(driver.title)driver.close()

If it prints “Google”, it means you have successfully run Selenium with Chromedriver on a Linux VM!

Bonus

If you are looking for a DockerFile, you’re in luck. Hope this helps!
Selenium for Test Automation — Yay or Nay?
By: Renju Jose
Tags: Selenium, Regression Testing, Software Testing, Automation Testing Tool

In this article, we’ll look at some of the aspects that led to the widespread adoption of Selenium as the de-facto standard for web automation. We’ll also check the scope for advancement and improvement in a Selenium-based test framework.

Selenium — Automation Testing tool

“Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes but is certainly not limited to just that. Boring web-based administration tasks can (and should!) be automated as well.”

As SeleniumHQ states above, the primary use of Selenium has been to automate Web Applications for Testing purposes, and it has been doing the same for a decade or longer. From Selenium RC and Selenium IDE, it has grown to a full-blown Web Automation library that supports multiple browsers and Platforms.

Selenium has been around for a long time, and it is, without a doubt, a great tool. It has been serving Automation Testing needs of testers worldwide for quite a long time.

Pros

1. Open Source/ No Licensing costs

Selenium has no upfront licensing costs or payments to be made for its use since its freely available. It is also an open source project which allows for extension and modification of the base framework for one’s personal or professional usage.

2. Language-independent

Devs or QA Analysts working on a particular language don’t have to learn a new language just to start automating their tests using Selenium. Selenium provides official Language bindings with popular languages such as C#, Java, Ruby, JavaScript, PHP, and Python in addition to the unofficial language bindings available for other languages.

3. Third-party Integrations

Selenium does not restrict a QA’s choice of reporting tools, build systems or any other aspect of their development/testing stack. It integrates well with popular tools such as SauceLabs, Selenium-Grid, Extent, JUnit, and others.

4. Parallel Testing

Selenium supports parallel test executions by integrating with Selenium Grid which helps to considerably bring down the feedback time for test executions.

5. Cross Browser and Platform-independent

Selenium supports all popular web browsers such as Google Chrome, Firefox, Safari, Opera, etc. and works on all currently available operating systems platforms such as Windows, Mac, and Linux.

6. Community Support

Selenium is backed by a vast community of developers and testers who actively contribute to the tool and documentation.
How to scrape news headlines from Reuters?
By: Aryan Soni
Tags: Selenium, Python, Data Science, Web Scraping

Photo by AbsolutVision on Unsplash

Web scraping can be a complicated task as some websites block you from scraping if you just use libraries like beautifulsoup in python and traverse through html. You need task automation library such as selenium so that you can scrape tons of headlines as this software clicks on load more button for you. It is really interesting to see how simple code in python opens the browser, scrolls down, clicks and scrape the data in a jiffy.

So lets start!

First we get the url of the reuters you are planning to scrape for eg. the url is https://www.reuters.com/news/archive/technologynews?view=page&page=6&pageSize=10

Now you need to see how yo can load more articles, each page have seperate ways to load more item and they are: Scrolling down, clicking on page number, clicking on next page or click on load more.

We can see that the page redirects to other page when you click on earlier

We can see that the class name is control-nav-next and the each news article has the class name as story-title. Now we have done the research on how the page works we can start coding in python

Before we start coding i want to tell you that we would be using selenium for this purpose. Selenium is a popular open-source web-based automation tool which is used for testing as well and we will be using the power of selenium to scrape headlines.

Lets jump to code

Start by installing selenium using: pip install selenium

Now create a seperate project and make sure you downloaded chromedriver. If not search for it on google and download latest version. If you download old version problems will start arising and you will not be able to debug it as it happened to me. So make sure you download latest chromedriver
Best Selenium Training Course Online
By: Bittu Kumar
Tags: Selenium, Best, Online, Course, Training

best-selenium-training-course-online

What is Selenium?

Selenium is an open-source tool that is used for automating the tests carried out on web browsers. It is a portable automated software testing tool for testing web applications. We can say that It is not just a single tool but a set of tools that helps testers to automate web-based applications more efficiently.

Selenium IDE is an easy to use tool for any person and which is designed to work with Firefox. It uses Firefox plugin with the graphical user interface, this GUI will allow the user to record test actions and replay. It also supports to write/run test cases without learning scripting languages as well as to all common languages such as Java, C#, Perl, Python, Ruby and Groovy scripts.

Benefits of Selenium Certification

Selenium is one type of freeware and open-source tool.

2. It supports multiple languages like Java, Perl, Ruby, Python, C#, and JavaScript.

3. It supports multiple operating systems like Windows, Mac, UNIX, and Linux.

4. Selenium supports multiple browsers like Chrome, Firefox and Internet Explorer.

Read more click here
Web Scraping and Price Analysis on Hepsiburada -via selenium and numpy
Tags: Selenium, Price Analysis, Web Scraping, Numpy

I was trying to decide which budget phone to buy, but also find the best price/value ratio. So I decided to remember an old tool of trade, web scraping. So I came up with a tool , consisting of 2 python codes for analyzing prices of all products of a category or a filter result , in-between the different sellers and their prices for the same products, and hopefully detect some cut-price sellers

I’ve shaped the xpath parameters for hepsiburada.com , the biggest retailer in Turkey. The results have been promising so far. On the phone category, whenever I find a seller that’s selling %20 cheaper than the “mean price”, it was almost always far more better than the best price in the other 2 biggest marketplaces, gittigidiyor.com and n11.com. Later, I generalized the code , so that it can take any category or filter result URL as an input parameter and output the url list of products, ready to be input for the analyzer script.

So, in final from , the first code outputs a list of product urls , and the second code outputs a CSV file with the mean and median prices of that product and the percentage of the minimum price compared to the mean and median.

If you would like to give it a try , you can find it on github . you would need python3 itself, 2 modules on python3 and the chrome driver for the chrome browser you will be using , which can be found here. Later you would need to change the CHROME_DRIVER_PATH in the code to show the driver’s path. I might be doing some more improvements for generalization and output formatting

The bash commands below should be helping you try it out for yourself, given that you have downloaded the appropriate chromedriver and replaced the path accordingly
Set up and run your Cucumber and Selenium web tests on Gitlab-CI within minutes
Tags: Selenium, JavaScript, Software Testing, Quality Assurance, Cucumber

Set up and run your Cucumber and Selenium web tests on Gitlab-CI within minutes spriteCloud Mar 14, 2019·3 min read

Build great software faster! CI/CD is hot and is still gaining interest. Test Automation is obviously essential within any Continuous Integration setup in order to get test feedback often and fast. Within the test automation groups here at spriteCloud we have seen a lot of different setups. We have been integrating test automation into CI tools such as Jenkins, TeamCity, Travis, TFS, Circle CI, Bamboo and Gitlab. Out of all the solutions we are a big fan of Gitlab, especially since it is making the integration of test automation a breeze.

.gitlab.yml config file for your Selenium tests

So without further ado, let’s give you the complete code you need to run your tests within a Gitlab CI:

Wait, what? I only need a configuration file? That’s it? Yes!

There are some prerequisites of course:

Have the Gitlab runner executor set to ‘docker’ (for this example)

use your selenium_remote_url environment variable within your test scripts. The selenium chrome service we started is accessible through: http://selenium__standalone-chrome:4444/wd/hub/

Let’s make the CI setup a bit more efficient though by adding some caching:

Configure your Selenium remote URL

The last thing to keep in mind is to include the selenium_remote_url in the test scripts. Here at spriteCloud we are using our open-source gem lapis-lazuli in order to quickly setup Ruby, Cucumber and Selenium test automation suites. This is how you integrate the URL within such a test suite:

But of course you can use selenium-webdriver directly as well if you want:

Want to try this out yourself in Gitlab? We have setup an example Gitlab project where you can see it in action! Gitlab Cucumber Selenium CI test setup Enjoy!

Further reading:

If you want to get started with test automation using ruby and cucumber you can read Getting started with test automation: Cucumber with Lapis Lazuli.

Thanks for reading. If you enjoyed what you just read and/or found it useful, let us know by clicking the 👏 below!
Acquire Selenium Training Certification for a Successful Career
By: Technosoft Academy
Tags: Selenium, Courses And Training, Selenium Test Automation, Software Testing, Training

Selenium training has got tremendous scope in the future of automation. It is a software-testing platform built for web applications. Today, it is instrumental in testing the functionality of user interface of various websites.

The training will basically help the students to learn the concepts from scratch. Various business enterprises are looking out for candidates who can be responsive to challenges of software industry.

With the help of selenium testing training online, every software expert can decipher the errors. No wonder that there is a huge demand for professionally trained and certified candidates who are well versed in selenium.

The demand can be attributed to the fact that selenium experts can reduce the burden of manual labor. With hundreds of software and mobile app launch every year, this is crucial.

In USA, there is a massive job vacancy for market-ready students. This is possible with the help of selenium testing training online that is both cost effective and involves little amount of time.

Market analysts have predicted that IT jobs will expand by 22% and if you have a certificate, you can fare better. The demand for software developers will be most powerful due to the rapid changes of a tech-based market. Since, software testing is a phase that is continuously in flux, you need to undertake selenium testing training online.

The same will help you to contribute the most to US job market and its requirement of creative minds. This sector will further enlarge 31% by 2026 for testers who has the highest potentials for troubleshooting a bug.

One of the crucial benefits for choosing the selenium training is development of relevant skills that would help you to cope out. One of the skills like configuration and management can help you create the best test automation strategy.

In case you need to distinguish yourself from rest of the automation engineer, this is the opportunity to do so. Apart from that, the training and various classroom activities will help you to stay updated with the recent trends, techniques, and tips to replace manual testers.

Compared to the other courses in the domain of IT like cypress, selenium certified candidate will get more job offers. According to experts, the candidate with Web-Driver skills will have more value and job growth. You can even select from the top-rated cities that pay the most to selenium testing trained candidates.
Run Selenium Tests on Google Cloud : using Docker, Kubernetes, Zalenium
By: Jayakrishnan Kunnath
Tags: Selenium, Zalenium, Automation, Google Cloud Platform, Kubernetes Cluster

Hey Alien’s, In this blog let’s see how to run selenium tests in GCP cloud environment using Kubernetes.

What is Docker and Kubernetes

Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management.

Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers.

Kubernetes

Set-up Kubernetes in GCP

Create a Kubernetes cluster

Kubernetes Engine -> Clusters-> Create

Create Kubernetes cluster

2. Connect to the cluster using “Cloud Shell” : execute below commands

kubectl create namespace zalenium

kubectl get namespace

helm repo add zalenium-github https://raw.githubusercontent.com/zalando/zalenium/master/charts/zalenium

helm search repo zalenium

helm install my-release — namespace zalenium zalenium-github/zalenium — set hub.serviceType=”LoadBalancer”

— set hub.basicAuth.enabled=”true” — set hub.basicAuth.username=”USERNAME” — set hub.basicAuth.password=”PASSWORD”

kubectl get service my-release-zalenium — namespace=”zalenium”

3. Access Zalenium

Now all set,we can access the Zalenium Grid and dashboard portal using following URLS.

Zalenium Hub : http://External IP_Cluster/wd/hub/

: http://External IP_Cluster/wd/hub/ Video : http://External IP_Cluster/grid/admin/live

: http://External IP_Cluster/grid/admin/live Dashboard : http://External IP_Cluster/dashboard/#

3. Dive into code

Time to Rock N Roll!!!

Zalenium grid is ready now we can access same using code, Here is an example with Java.

4. Result :

Video of execution

So That’s all i have in this, Kubernetes will take care of auto scaling and down of the zalenium container as per the request it receives.

Thank you for checking this out!! Please comment if you get any queries.

Please share and subscribe!!
TestNG setup using Maven for Selenium
By: Knoldus Inc.
Tags: Selenium, Maven, Selenium 4, Datadriventesting, Testng

Definitions

TestNG (Test Next Generationis) is an open source testing framework. TestNG is inspired from JUnit and NUnit. It is an improvised framework which makes it more powerful and easier. It provides the developer to write more powerful and flexible tests with the help of easy annotations, sequencing, grouping and parametrization. We can use TestNG for generating reports. The reports generated will be of two types pass and fail. If we wish we can run any of the report anytime.

Maven was created to have a standard way in which Projects can be built. One of its powerful features is dependency management. IntelliJ IDEA has in-built support for Maven.

Isn’t it great to use TestNG and Maven together Lets get started with TestNG setup using Maven for Selenium

Prerequisites

Java

Inellij

Don’t worry if the prerequisites are not installed. You can refer to my previous blog.

How to import Maven project in Intellij IDEA?

Go to File > Project

Select Maven and Project SDK to java package and click next.

Add GroupId, ArtifactId and click next. We should add a project name after clicking next. We are ready to implement.

Go to Pom.xml and add dependencies for TestNg, Maven, Selenium

We have to add selenium Java in Pom.xml from the below external link.

https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java

For TestNg go to the below link.

https://mvnrepository.com/artifact/org.testng/testng

Copy the dependencies and paste in POM.xml. Now you will get option to import the dependencies. Do that.

How to use TestNg annotations ?

If you are not aware of the format used for TesntNg, stay calm and use the below format.

@DataProvider (name = “name_of_dataprovider”)

public Object[][] dpMethod() {

return new Object [][] { values}

}

Good going. Now you have to create a java project and implement the below example. Run the code as TestNGTest

import org.testng.annotations.Parameters; import org.testng.annotations.Test; import org.openqa.selenium.By;

public class MyFirstTestNg { @DataProvider (name = “data-provider”) public Object[][] dataMethod(){ return new Object[][] {{“FirstName”}, {“SecondName”}}; }

@Test (dataProvider = “data-provider”) public void test (String val) { System.out.println(“Passed Parameter Is : “+ val); } }

Run the above example as Run As -> TestNG Test.Run testing.xml and check the output.

<suite name = “Suite”> <test name=” Test”>

<classes> <class name = “Code.MyFirstTestNg”>

</class> </classes> </test> </suite>

You should run testing.xml and after running you check the output

Congratulations we have completed a data driven program with the help of TestNg hence TestNG setup using Maven for Selenium

References:-
Selenium or Selenide?
By: Davit Danelia
Tags: Selenium, Test Automation, Programming, Automation, Java

Why TBC Bank is using both of them?

I want to introduce you to Test Automation in TBC Bank, specifically about how we are supporting TBC products with test automation, what frameworks are we using and why?

As we all know, big products needs big support for a perfect performance, it’s getting harder to catch regression bugs in any level of production while your project is growing by the day, new functionality, components, logic, rules, features, we need all that without any bugs, for that every team has not only manual testers but also Test automation engineers for any kind of automation testing, today i gonna talk about UI automation testing frameworks that most of TA engineers are using nowadays and specifically about Selenium and Selenide and why sometimes we prefer Selenide over Selenium.

First of all lets talk about Selenium, writing a successful UI Selenium tests can be challenging at times. AJAX requests, timeouts, code duplication, and dynamic content pages are only a couple of the challenges that Automation Engineers face. Selenium Webdriver is one of the most commonly used UI testing tools not only in TBC but in every other companies too. however it is important to realize that Selenium is a low-level framework library for handling browsers. Trust me, I have been using Selenium for a long time and I can assure you that it is not fully optimized for handy test writing.

Selenide is a framework based on a web driver from Selenium. Selenide has been developed for automatic testing as a simpler and easier option. It has many special characteristics, for example:

Smart waiting ( more intelligent waits )

) Selenide wrapped WebDriver

Comfortable methods

Ajax support

Automated screenshots

Customization

Let’s find out how Selenide streamlines the tasks of the testers and what benefits Selenide has over Selenium.
The hidden classes no one knows about which help you locate the elements in Selenium.
By: Abhishek Dhoundiyal
Tags: Selenium, Byidorname, Locators, Byall, Bychained

The hidden classes no one knows about which help you locate the elements in Selenium. Abhishek Dhoundiyal Follow Apr 7 · 2 min read

There are 3 Selenium classes that are not so popular and so many people even hadn’t heard about them. These classes can come in handy in our day to day tasks.

These three gems are:

- ByAll

- ByChained

- ByIdOrName

1) ByAll

Package org.openqa.selenium.support.pagefactory

org.openqa.selenium.support.pagefactory.ByAll

Implementation:

We have to call the constructor:

ByAll​(By… bys)

Arguments: Pass the Multiple locators separated by a comma.

The mechanism used to locate elements within a document using a series of lookups. This class will find all DOM elements that match any of the locators in sequence, e.g.

driver.findElements(new ByAll(by1, by2))

will find all elements that match by1 and then all elements that match by2. This means that the list of elements returned may not be in document order.

— — — — — — — — — — — — — — — — — — — — — —

2) ByChained

The mechanism used to locate elements within a document using a series of other lookups. This class will find all DOM elements that match each of the locators in sequence, e.g.

Package org.openqa.selenium.support

org.openqa.selenium.support.pagefactory.ByChained

Implementation:

We have to call the constructor:

ByChained​(By… bys)

Arguments: Pass the Multiple locators separated by a comma.

driver.findElements(new ByChained(by1, by2))

will find all elements that match by2 and appear under an element that matches by1.

Code:

driver.findElement(new ByChained(By.id("email"), By.tagName("input"))).sendKeys("abc@gmail.com");

— — — — — — — — — — — — — — — — — — — — — — — — —

3) ByIdOrName

This helps the driver to locate an element either by name or by id.

Package org.openqa.selenium.support

org.openqa.selenium.support.ByIdOrName.ByIdOrName(StringidOrName).

Implementation:

We have to call the constructor:

ByIdOrName​(java.lang.String idOrName).

Arguments: Pass the Name or ID.

This method first tries to find the element using ID and waits for the element to locate and if it is not able to find the element with ID then only it tries to locate with the Name.

Code:

driver.findElement(new ByIdOrName("by1")).click();

WebDriver.Support (in WebDriver.Support.dll) Version: 3.1.0

Note: Complete explanation will come in the next part.
End to End Testing React app with Selenium WebDriver Chrome PyTest and CircleCI
By: Yi Ai
Tags: Selenium, React, Pytest, Docker, Circleci

In order to create automated testing in a continuous integration environment, you will need to familiarise with a series of automation and testing tools for different situations. In this post, I’ll explain how to use Selenium Python implement and automate e2e testing for React app in CircleCI.

Overview

There are lots of tools and techniques that can be integrated in a CI environment. Here at Neami, our development team is mainly using nodeJs and React, but test team prefers to use python. thus i worked out this design to make both 2 teams happy ( testers would write test case in Python rather than switch to javascript ).

We are using Selenium WebDriver Python,Chrome and we integrate them with CircleCI. The testing framework we are working with is PyTest, which makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.

The main techs we use are:

React

Selenium

Chrome

PyTest

CircleCI

Frontend

In this example, Frontend is to do list app built with React, you can find source code on this repository.

Docker

There are lots of benefits to use docker, I’m not going to go through of the benefits here. Let’s create a base Dockerfile to support Selenium, Chrome and React:

Next, create a Bash script docker-entry.sh.

#!/usr/bin/env bash set -ex cd pytest nginx pytest -s --url http://localhost

Selenium and PyTest

Now, It’s time to look at Selenium test, create conftest.py (modules auto-discovered in test directories)

$ mkdir pytest

$ cd pytest && touch conftest.py

Copy following code to conftest.py

Now, create pytest/test sub directory and test case file:

eg. I created a test_todo_list.py:

We are ready to run the script! Run the command:

$ docker build --no-cache -t e2e-test .

$ docker run -p 80:80 --rm e2e-test

The tests are being run and you can see the results:

CircleCI Integration

Now, Create config.yml in .circleci directory at root of project. eg. config.yml should like

version: 2

jobs:

build:

machine: true

working_directory: ~/todolist

steps:

- checkout

- run: docker build --no-cache -t e2e-test .

- run: docker run -p 80:80 --rm e2e-test workflows:

version: 2

frontend:

jobs:

- build:

filters:

branches:

only:

- master

CircleCI Workflow Job being run for every commit, you can see the result in CircleCI dashboard:

That’s it!

You made it all the way until the end! No need setup local env, no softwares and tools required, you don’t even need docker, just push your code to Github, CircleCI runs testing job after every committing.

If you get stuck along the way, feel free to check out the code on this Github repo.:

Learn more

Join our community Slack and read our weekly Faun topics ⬇
Creating dataset— Scraping Marathon Images Using Selenium and BeautifulSoup
By: Kapil Varshney
Tags: Selenium, Beautifulsoup, Programming, Web Scraping, Computer Vision

Where to get the dataset

Okay, I read the intro blog. It sounds great! But, how do you get started? You need data, right?

Right. To be able to even try any one of the methods to recognize the bibs, that were described in the intro blog, we need data — marathon images, and many of them.

One obvious way was to go to a running event myself and photograph a few (at least 1000, I planned) images using my DSLR and phone camera. But, none of the Sundays I could get myself to get up at 4 am, ride my bike for an hour in the cold Bangalore morning and reach a running event to cover it. On one of the days that I did, it was to run myself.

The other way, I figured, was to get the images from the race organizers. I contacted the team of Kaveri Trail Marathon (KTM), a beautiful run I had participated at recently. KTM team was happy to help and shared with me the contact details of MyRace, their tech partner for the race. I tried to contact MyRace and to get to talk to the concerned person to discuss this project and ask for help, many times, but in vain. That’s when I thought of trying to scrape the images from their website. Jugaad! It did take a while to figure out the right hacks to do so, but I got it working eventually.

Exploring the website to scrape

Nice, but how do you figure out how to scrape MyRace website for the images? I mean, a website is a big jungle of HTML/CSS/JS. How do you navigate through it, identify the relevant parts and extract the images?

With patience and persistence, my friend!

To get started, there are a few prerequisites:

Understanding of basics of HTML and how a website is structured Experience with BeautifulSoup, the python library for web scraping. Know-how of Selenium drivers and its Python API A bit of smartness will help as well ;-)
How to Scrape Multiple URLs with Python: Tutorial
By: François St-Amant
Tags: Selenium, Beautifulsoup, Programming, Web Scraping, Data Cleaning

Looping over the page number

This is the simplest, most straightforward way of scraping multiple pages. Let’s begin by looking at the end of the URL we are scraping the hostels from (full URL available at the end of the article):

We see that for the first page, we have page=1. For the second page, we would have page=2, and so on.

Therefore, all we need to do is create a “for” loop where we change the very last number. For each page, the loop will collect the information we specified.

Here is the code to collect the distance from city centre, the price of a dorm bed, the price of a private room and the average rating given by previous customers for all the hostels found in the first 2 pages of the website.

I use selenium here because the hostelworld pages are JavaScript rendered, which BeautifulSoup cannot handle.

I scraped the “price-title 5” element because this element allows us to know whether the price is for a dorm or a private room.

The sleep function is useful to control the rate at which we make requests to the website server (to avoid slowing down the servers), but it’s also useful to make sure selenium has found the information we want before it keeps going.

Normally, we would move on to cleaning the data to make it usable, but I will do this at the very end with the last method. Let’s move on to the next method.
Setting up a scraping server on Digital Ocean — Part 1
By: Abhishek Chaudhary
Tags: Selenium, Development, Scraping, Deployment, Python

We’ll be using Digital Ocean Droplet to set up our scraping server.

Step One: Registration

In order to run the server, you’ll need a DO account. So head over to this link and get yourselves a shiny new account.

Once you’re registered, visit the Create Droplet page from the top menu.

Step Two: Creating a Droplet

Choose an operating system, we’ll be working with Ubuntu

For the server, we’ll be picking the smallest size server. We don’t need anything bigger right now.

No block storage is required for now.

Select data-centre region closer to you.

We don’t need any additional options for now so skip ahead.

SSH Keys

When you’ll click on new SSH key you’ll be prompted with the following window

Here you need to enter the public key for you ssh private-public key pair.

Step Three: Creating and using a private-public key

SSH key — it has two parts, a public key that we give out to people and web sites, and then we keep a private key to prove that yes, we are who we say we are. That’s the super short way of explaining it.

Windows , follow this guide to generate an SSH key.

, follow this guide to generate an SSH key. OS X, you could follow this guide but really, just open up the command line and run the following command.

After the public-private key pair is generated, enter the public key in the dialogue box and hit add ssh key

Finishing Up

Change your hostname and limit the number of Droplets to 1.

Setting up your server

Now you can move on to setting up your server

Support Hacker Noon via this link:
Multilogin 3.1.1 Rise STABLE released
Tags: Selenium, Automation, Multilogin, Browser Fingerprinting

Proudly presenting you the culmination of the third Multilogin line — version 3.1.1 Rise STABLE. It combines all innovation of the Rise line with unrivaled stability and comfort of work. Just a quick reminder of what the third version brought to you:

Statistically valid browser fingerprints based on real internet user data

Across the board improvement of fingerprint masking

Filling database with fresh browser fingerprints every minute

Availability of all browser features such as password saving, browser history, and bookmarks

Improvement of work with numerous types of proxies and SSH-tunnels

Automation features and of course a long-awaited API v2

In the version 3.1.1 Rise we polished all mechanisms and brought them to perfection. All significant problems were fixed.

User migration to 3.1.1 Rise

Following the release of a stable version we disable older BETA versions. This will ensure everyone is using a bug-free software. This time besides disabling beta versions we will also disable versions 1.4.x and 2.x with an exception of version 2.3.2 as we announced earlier.

Disabling old versions will allow us moving forward at a faster pace. We will be delivering you more astonishing features further improving comfort and speed of your work.

The full list of changes in 3.1.1 Rise
Running Python Selenium script out of Linux
By: Roger Z
Tags: Selenium

Tools you will need:

Selenium python package

apt-get chromium-chromedriver

set-up with appropriate options.

#Premise#

I was tasked to generate a simple report card everyday for finance department. Even though it was a simple action — open browser, open several tabs and input search string to record down numbers and compile it to a small report card, I find it incredibly dull and soon become tedious. So I set myself out to write a python script that can generate the report card for me.

Better yet, the script will run on cron tab and automatically send through a chatbot so I don’t have to do anything at all.

At first, I’ve written the script locally, and it works nice enough.

A simple trick to avoid a lot of hassel in setting up path is to to put the chrome driver in python’s /Scripts folder. (http://chromedriver.chromium.org/)

Since I’m using conda’s virtual environment, the full path looks some like this:

C:/Users/[Administrator]/AppData/Local/conda/conda/envs/[envname]/scripts

By placing the chrome driver here, I was able to run the selenium without any further troubles.

class selenium_driver():

def __init__(self):

chrome_options = Options()

chrome_options.add_argument(“ — headless”)

self.driver = webdriver.Chrome(options=chrome_options)

Interestingly, when the script is finished testing and uploaded to linux, it was not so straight forward. Given the linux server was an ec2 instance from Amazon, it requires a few extra setup in order for the script to run.

First, the ec2 needs to install chromium program, there are two, both should in-theory work, but I’ve chosen the package called chromium-chromedriver

apt-get install chromium-chromedriver

After installtion, both chromdriver and chromium-browser shoud be in /usr/bin path. *in order for the chrome driver to work with selenium, these two packages should be added to path if you choose to put them in non-path directory.

This should have worked, but the script crashed instantly. After some further research, I’ve realised that the chromedriver and chromium will not work in this set-up as the EC2 does not have GPU and cannot run as sandbox.

Therefore adding the following options:

chrome_options.add_argument(“ — no-sandbox”)

chrome_options.add_argument(“ — disable-gpu”)

The result is a nice autonomous script which I have put on crontab to run on a set time everyday.
Silicone Labs — Software Quality Assurance & Job Outlook
By: Ahmet Can
Tags: Selenium, Qa Automation, Testing

Software quality assurance is one of the most critical parts of software development whether you’re developing an application or an end-to-end system.

Manual Vs. Automation Testing Manual testing is a method used by software developers to run tests manually whereas the method takes automation tool support to execute the test cases known as Automation Testing. Automated testing is usually good for large and detailed projects.

What is Automation Testing? Automation testing is known as the process of using a computer program to execute systems or user transactions against an IT system.Automated testing tools are capable of executing tests, reporting outcomes and comparing results with earlier test runs. Why Use an Automating Testing? When a software tester manually checks a system, she can make a mistake, especially when an application contains hundreds to thousands of lines of code. Automation helps the QA team avoid these human errors in application testing and executes checks in a faster time frame than if it were done in person.

What are Some Automation Tools? Open source test automation tools and frameworks include Selenium, Robotium and Cypress. Selenium can automate and run test parameters across multiple web browsers and in various programming languages — such as C#, Java and Python.

Software QA Engineering Jobs Outlook & Trends According to the Bureau of Labor Statistics, Software QA Engineering Jobs are growing very rapidly with an estimation of 100,000 new positions set to be created through 2022. Companies’ demand for skilled QA engineers are growing and they are taking their positions as an integral part of any software development platform.
Simple Unfollower Bot with InstaPy
Tags: Python, Instagram, Automation, Selenium

Hey everyone! I’ve got a short post for you today on how to create your own unfollower bot to search for and unfollow people who don’t follow you on Instagram, and at a pace that will not get your account banned.

I wanted to make my own bot for this even though there are plenty of apps that do this. However, you can never be sure how secure these apps and websites are or what they do with your login information. So to avoid getting hacked, with a little knowledge of Python, you can make your own bot. With a project called ‘Instapy,’ by TimG making your own unfollower bot is actually pretty easy.

Read below for the step-by-step:

Step #1: Pip install Instapy

pip install instapy

Step #2: Write the code

from instapy import InstaPy session = InstaPy(username="your_name", password="your_password") session.login() session.unfollow_users(amount = 10, nonFollowers=True, style="FIFO") """unfollows 10 people, but only people who do not follow you, in the order of First In, First Out (FIFO)""" session.end()

That’s it! Run the program and watch it work. If you don’t give it any specific instructions, InstaPy will sleep periodically to avoid doing too much activity too quickly, but you can increase or decrease the amount to unfollow, and you can change the style to LIFO, which would instruct the program to unfollow the earliest people you followed, instead of the most recent. You can even do something like this (Caution: Add this line before the unfollow_user() method):

session.set_dont_unfollow_active_users(enabled=True, posts=1)

With this line, people who have interacted with your most recent post will not be unfollowed, even if they are not following you. You can specify how many posts that person must like in order to be classified as an ‘active user.’

Step #3 (Optional): Set Quotas

Add the following code before session.unfollower_user():

session.set_quota_supervisor(enabled=True,

sleep_after=["unfollows", "server_calls"],

sleepyhead=True,

stochastic_flow=True,

notify_me=True,

peak_unfollows_hourly=20,

peak_unfollows_daily=250,

peak_server_calls_hourly=200,

peak_server_calls_daily=2500)

You can do all sorts of things with Instapy. Explore the documentation to build a bot that will suit your needs.
WinAppDriver with Selenium Grid
By: Charlene Liu
Tags: Selenium, Automation, Software Testing, Winappdriver, Appium

What WinAppDriver can help you?

When you know much about selenium web driver, running another automation testing for windows application with winappdriver should not be a problem for you. And the best part for the automation testing, you have various programming languages to choose from.

In my project, I need to run automation test scripts on multiple windows based machines. In the past, we developed test scripts and performed them on each machine, so we had to collect the test result one by one. With the help of Appium, I get to try selenium grid to run multiple sessions on different machines in the meantime.

But in our use case, even though I can upgrade test machines to windows 10, most of the UI elements in our legacy windows application have difficulty to be located by inspect.exe. (something like, there is no unique className on the same page. Not sure if it’s caused by the way it was designed. ) Since we currently don’t have plan to modify the legacy program, the worse case for me is probably just run the scripts to send keystrokes for operating purpose and find other methods to verify the test results. (for instance, verify metadata created by certain operation)

Selenium Grid architecture

Every installation details can be found here. I am using MStest to write my c# test scripts with .net framework 4.72. First step is to nuget all the necessary selenium, appium and windriver packages. I include a .test.runsettings file to configure the location of testing data and report. I am also able to run it in command line by vstest.console.exe after build.

For setting up selenium grid, I use my local machine with JDK installed as the hub with jar file running. And I install node.js on my test machine. It will come with npm after installation. But using npm to install appium always get me older version of windows driver, I have to correct it by npm uninstall the old one and install the new one back. And then running appium with a JSON config file will register my node.

It is amazing to see those test scripts running on multiple windows based machines in parallel. It indeed saves our time to do the testing and operation work.