fabiochiu/medium-articles huggingface dataset:
Windows Server and Linux Monitoring Tool with NETDATA
By: Umut Ergün
Tags: Windows Exporter, Netdata, Monitoring, Wmi, Monitoring Software

Netdata Monitoring tool

Hi everyone;

Today we will try to install netdata in our linux and windows servers.

After than we will design a new custom HTML file for netdata tv.

Netdata is a very useful application and very light for memory usage and Cpu usage for monitoring. You can monitor everything in real time and free.

I will show you the basic installation today. For more detail please check the official web page.

Now let we start

First we need a linux machine. In this scenario I used a Ubuntu 20 LTS distribution.

This is my configuration but you can increase or decrease this options

RAM : 4 GB / CPU : 2 VCore / Disk : 50 Gb

Please ssh to the ubuntu server with root or root permission and run the command.

Please check the official installation page for more details.

After installation is finished everything is done. Now netdata is working. Let the check with your browser

Open your web browser and write the following address http://your_ubuntu_server_ip:19999

If you see the monitoring details in your browser then it says net data installation is successful for ubuntu.
Cloud Egress visibility in Aviatrix Monitoring-Only Mode
By: Tomasz Klimczyk
Tags: Visibility, Security, Cloud, Monitoring, Aviatrix

DC to Cloud

Back in the day, when your application resources were residing in an on-prem DC, you did not really need to worry about the security for the ingress/egress to the Internet. There would be a security team with a wide range of tools and services monitoring and protecting your workloads — NetFlow, FWs, visibility tools monitoring the traffic inline and out-of-band. It was all possible because of physical and logical separation of the various layers of your DC network.

In the Cloud it is different — the Internet is just one hop away from your resources (VMs, containers), and protecting and monitoring that Internet traffic is not that simple any longer. You do not have the benefit of inserting a physical tap on the switches and routers through which your packets are flowing. How do you monitor the egress traffic then, to make sure your resources are not compromised? Or how do you provide insights into breach that has happened? How do you provide that information when an audit comes? How do you make sure your cyber security teams have enough information to actually work with?

What are your options of monitoring your cloud traffic egressing to the Internet?

Have you ever been wondering what your VMs are reaching out to as they go via the Internet Gateway, IGW, NAT GW, or simply the default Internet egress in any of the major clouds (AWS, Azure, GCP, OCI, Ali) ?

The native options in these cloud providers are quite limited and the best they can offer are some form of flow logs (VPC Flow Logs, VCN Flow Logs, NSG Logs). There are a few challenges though:

- generating the information (continuity of the monitoring, sampling of the packets, granularity of selection what to monitor — entire VPC or just a VM?)

- receiving and consuming the information (read the text file with the logs, build your own analyzer or leverage a 3rd party solution — would it be mulit-cloud though?).

Aviatrix Monitoring-Only Mode for Egress

Aviatrix Multi-Cloud Networking and Security Platform offers a number of services to help your connectivity and security in the cloud. However, you can also use it only for the purpose of gaining visibility into your egress traffic.

The deployment model is simple — you install Aviatrix Controller, CoPilot and Egress GWs in each cloud network which you want to monitor, as in the diagram below:
How to Deploy Grafana on Debian-like Linux in AWS using Ansible.
Tags: AWS, Grafana, Monitoring, DevOps, Ansible

Step 3.

Install Ansible on your local computer.

Step 4.

Clone this repo:

Step 5.

Modify ./tutorial-aws-grafana/ansible/hosts file:

hosts

You should add your pem file location after “ansible_ssh_private_key_file= ” (line 3) and replace 54.77.154.230 to your AWS instance public IP (line 6)

Step 6. Run Ansible playbook

$ ansible-playbook -b grafana.yml

In this playbook, we install apt-transport-https, which is required by grafana, then add grafana repository and gpg key, install grafana-server and finally start service and enable it after an instance restart.

Ansible playbook

Ansible can show a warning. Nevermind.

Step 7. Logging in for the first time

To run Grafana open your browser and go to http://your-aws-instance-ip:3000/. 3000 is the default HTTP port that Grafana listens to. There you will see the login page.

Grafana login screen

The default username is admin and the default password is admin. When you log in for the first time you will be asked to change your password. We strongly encourage you to follow Grafana’s best practices and change the default administrator password.
Sarkis Khoury’s profile is secured to the strictest privacy by VirtualU Privacy-Shield.
By: Sarkis Khoury
Tags: Privacy, Data, Sarkis Khoury, Monitoring

Sarkis Khoury’s profile is secured to the strictest privacy by VirtualU Privacy-Shield. Sarkis Khoury’s data privacy settings have been set to the strictest settings and to abide by this website’s Privacy Policy and Terms & Conditions. You’re activity here will not be shared with 3rd parties unless it is necessary for this website to provide service to you. Get your own Live Social Media Monitoring, Data Privacy Protection or have a Guru help secure your “Virtual You”. https://virtualu.net/shop/?wpam_id=17

tags: privacy, shield, personal, business, data, social media, monitoring, software, service, tool, cloud, squad, guru, company, dashboard, free, trial, US, EU, GDPR, certification, list, principles, requirements, self certification, compliance, suspension, onward transfer, right to be forgotten, ORM, online, reputation, management, company, business, website, software, do it yourself, service, bank account, credit score, attorneys, doctors, dentists, restaurants, students, children, family, for dummies, contact, near me
Monitor firebase cloud functions in realtime
Tags: Firebase Cloud Functions, APM, Monitoring, Observability, Serverless

Firebase supports different types cloud functions like http, timer, firestore, realtime database triggers etc. Only way to find if something goes wrong in cloud functions is checking logs in firebase/GCP console.

In order to monitor the performance issues and errors occurring in cloud functions, we have developed agent module and that collect metrics in realtime and shown in https://nextapm.dev dashboard.

creating cloud function monitor in https://nextapm.dev

Install agent module in your application.

npm i nextapm

Include the following snippets in main file of cloud funtions.

// this should be in the first line

import { config, setFunctionName } from 'nextapm';

... your app imports ...



config({

licenseKey: 'xxxxxxxxxx',

projectId: 'xxxxxxxxxx',

});



...your cloud functions...

functions.firestore.document(path).onWrite(async () => {

setFunctionName('SendMail'); // call in all cloud functions

....function logic....

})

To track exceptions, use following agent API.

import { trackErr } from 'nextapm';



try {

... your app code ..

} catch (err) {

trackErr(err);

}

If setFunctionName api used then mentioned function name will be shown in transaction name. If not then firebase operation functions names will be shown.

Nextapm Dashboard

Please check it out and give feedback about this.
A/B TESTING AND UX ROADMAP, DASHBOARD TEMPLATE FOR MONITORING
By: Elisa Greef
Tags: Template, Ab Testing, Monitoring, Dashboard, UX Design

Photo by Firmbee.com on Unsplash

Hello 🙋‍♀️

Some weeks ago I talked in a webinar about A/B testing and the tools I use to monitor my A/B tests. I hope it’ll help some of you struggling with their process !

As the only person in my (not so small) company managing the UX and the A/B testing process, I need to stay well organized and to make it easier I use different dashboards and tables to put all my data in.

The good thing about tables is that after a little amount of time creating it, you’ll gain so much more by just filling the column and lines. Also, you won’t forget any useful information and you’ll keep an history of your data and work process.

I made a template inspired of those dashboards, for you to duplicate and use it as you want 😀. You can find it here.

Monitoring UX inputs and A/B testing

(I tried my best to translate it well from french to english, but do not hesitate to tell me if something is not clear enough!). You can also use the process on another platform if you prefer like Trello or Notion, it’s up to you !

It’s really easy to use on a daily/weekly basis around your UX inputs and A/B testing. There is 3 tables :

The first one is about your UX inputs, what are you going to do, why and how.

The second one is about your A/B tests and how to monitor them.

Then the last one is about your tests result and keeping an history of it, easy !

You will find a use case in the template to show you the way I’m using it.

I hope you found this publication useful and wish you a very good day 😊 !
Elaisian raises €1m in latest funding round
By: Sara Collinge
Tags: Monitoring, Agriculture, Apps, Farming, Agtech

A new take on an old tradition

Try and think of an industry defined by its cutting-edge technology. Any industry at all. What springs to mind?

Whatever you chose, I bet farming didn’t feature highly on your list. While industries like manufacturing and software development are known for their fast-paced advances, it’s probably fair to say that we still think of agriculture as quite an antiquated industry.

Enter Elaisian: an Italian tech company determined to revolutionise farming with the help of AI. The company, which has so far raised €1.5m in seed funding, is a precision faming system that allows farmers to monitor the weather conditions in their olive groves, orchards and vineyards in real time and detect pathogens in advance before treatment is needed. With the aid of a handy app, farmers need only intervene when necessary, which enhances production processes and reduces annual costs.

Originally designed for use in Italy’s olive groves, the company’s tech can be used in any arable farming setting

So how does the tech work?

The company’s main product range at the moment comprises a weather station, climate alarm, decision support system and satellite alarm. The solar-powered weather station has a rain gauge and sensors to detect humidity and temperature, while the climate alarm alerts farmers by SMS on certain weather conditions, pre-programmed by the user.

The decision support system (the app) displays real-time weather data, pathogen monitoring and predictive alerts, and includes an activities section where farmers can keep a log of any treatments (e.g., fertiliser) they have used on the land and when. The company also uses images from the EU Copernicus satellite and sends reports on various indexes, including vegetation, water stress and food stress.

Global recognition

Not only has the company been named among the best 500 foodtech start-ups in the world, but its co-founders — Damiano Angelici and Giovanni Di Mambro — were named as among Forbes Italy’s talents under 30 for 2020. The duo launched Elaisian back in 2016 and the app is now used in 10 countries, 19 regions of Italy and 30+ towns in Spain.

Perhaps unsurprisingly, given that Angelici has a master’s in social media marketing, the start-up is bossing its SEO: a simple Google search brings up Elaisian’s website, LinkedIn, Instagram, YouTube and Facebook profiles —all in English, too.

What does the future hold?

The farming solution has huge potential for regional expansion. Over the next few years, smart tech will increasingly intersect with farming , especially in developing countries — which is where Elaisian could come in to protect against scenarios like droughts and failed harvests.

While agriculture has been around since time immemorial, it’s by no means perfect and hasn’t benefited from smart AI solutions — until now.
Data breach monitoring for emails
Tags: Security, Infosec, Monitoring, Data Breach, Email

Earlier on we at Badrap.io introduced an integration to Have I Been Pwned data breach monitoring for a single email address.

Today we’re back with multiple email monitoring!

Now you can add in all of your email addresses and get alert notifications on data breaches. Emails such as personal addresses you use at home or at work, or emails you’ve used to register to online services and mailing lists. Even the so called plus-addresses (me+service@domain) work beautifully.

Add in all of your email addresses and get alert notifications on data breaches.

You might also want to consider adding in your loved ones emails, like your mom or kids, with their consent of course. Badrap always sends verification requests to the actual email owners, asking for their agreement on adding the email to your list of email assets.

Badrap always sends verification requests to the actual email owners, asking for their agreement on adding the email to your list of email assets.

You’re always in control

It’s important to note that one can always retract their email address from Badrap.io’s email data breach monitoring. If you accidentally accepted or your situation changed, you can visit Badrap and subscribe to email removal. With a few clicks your email address gets removed from every user of the Badrap service. No hassle — You’re always in control.

I’m not gonna give my email inbox to anyone

By giving another person permission to monitor your email address for data breaches, you’re not giving access to your actual email inbox. You’re giving permission for the other person to get notifications when your email address is involved in a data breach. This way they can contact you and help with the aftermath.

Start today!

In addition to email addresses, Badrap also monitors for vulnerable devices and services with the IP addresses of the networks you use.

Visit Badrap.io to get started today!
Adding Linkerd 2.x data to your main Grafana dashboard.
By: Christiaan Vermeulen
Tags: Grafana, Linkerd, Monitoring, Prometheus, Kubernetes

Install Linkerd on your cluster.

Just before we start, I want to make it clear that linkerd install does nothing amazing. It merely outputs the Kubernetes yaml configs required to run Linkerd. This is why we pipe it into kubectl apply later.

Its always valuable to look through the output.

1. See what linkerd is actually deploying & doing

2. Possibly learn something about how the big guns are doing their configs ?

If you would like to inspect what linkerd is deploying, simply run

$ linkerd install >> linkerd.yml

This will insert the output from linkerd install into a file called linkerd.yml

If you open this file you will see a bunch of configs which linkerd will deploy

You 1 — Magic 0.

With that being said. Lets make sure the cluster is ready for linkerd

$ linkerd check --pre

The above command will ensure linkerd can be installed and operate on the cluster.

Now we install linkerd. Pay attention to the syntax here. All we are doing is piping linkerd yaml configs into kubectl apply with

| kubectl apply -f -

$ linkerd install | kubectl apply -f -

namespace/linkerd created

serviceaccount/linkerd-controller created

[...]

Inspect linkerd in the cluster

$ kubectl get pods -n linkerd

NAME READY STATUS

linkerd-controller-5dfbd756b7-snwpw 4/4 Running

linkerd-grafana-55784d8c8c-nmhdz 2/2 Running

linkerd-prometheus-64854cdffd-r5phv 2/2 Running

linkerd-web-7845d9cf6d-9m5x8 2/2 Running

Proxy linkerd’s dashboard to see what it looks like

linkerd dashboard

Linkerd dashboard available at:

http://127.0.0.1:63399

Grafana dashboard available at:

http://127.0.0.1:63399/grafana

Opening Linkerd dashboard in the default browser Linkerd dashboard available at:Grafana dashboard available at:Opening Linkerd dashboard in the default browser

Go to the address it spits out above and you should see something like this:

The Linkerd default dashboard

Now in the search bar enter the second address that got spit out. It ends with /grafana .

This will open linkerd’s grafana dashboard.
Sending metrics directly from Ceph to Influx
By: George Shuklin
Tags: Influxdb, Telegraf, Monitoring, Ceph, DevOps

Sending metrics directly from Ceph to Influx

After this sad bug in telegram (some Ceph metrics are missed for newer version of Ceph) I was delighted to find that Ceph (ceph-mgr to be precise) can send metrics directly into influx. I’m giving instructions for Luminous.

Configuration

There is one big WTF issue in this setup: it has a hidden dependency. If this dependency is not installed, no data will be send. It’s a python-influxdb . Install it before doing anything else on all mgr hosts.

After you’ve setup this dependency, enable influx module.

ceph mgr module enable influx

Restart all mgr services (if you have a normal installation, it’s at least three of them).

And provide a configuration to it:

ceph influx config-set {{item}} {{value}}

options list:

— username

— interval

— database

— hostname

— ssl

— verify_ssl

— password

— port

The ‘influx’ command will appear only and only if you have enabled influx module and this module can import influxdb client. If it can’t, it silently fails. Moreover, you must to restart all ceph-mgr instances after you’ve enabled the module.

After you’ve configured everything you can check if it works:

ceph influx send

You should get confirmation and two measurements should appear in influx:

ceph_daemon_stats

ceph_pool_stats

ceph_daemon_stats has per-daemon information. Depending on a daemon type it would be osd-related or mon related info (mgr is there too, and mds, etc).

ceph_pool_stats has pool-related stats.

Those are more than enough for most purposes of having ceph metrics. The single exception I find is a ceph health report which I gather as a normal metric, because it so nice to show on dashboards…
Priyanka Enters in Active Politics — Digital Marketing Blogs
Tags: Priyanka Gandhi, Monitoring, Politics, India, Congress

Monitoring

Priyanka Gandhi

Arrival of Priyanka Gandhi into the heart of the political battle is a double-edged sword.

With the eye on 2019 lok sabha elections, Congress played a game-changing move. Priyanka Gandhi enters in active politics and appointed as a general secretary of the eastern block of Indian National Congress’s in Uttar Pradesh (UP). Congress also appointed Jyotiraditya Scindia as general secretary for Uttar Pradesh West and KC Venugopal as general secretary of the organization.

The one who wins the heart of Uttar Pradesh gets the center. With almost 80 key seats coming from the state, UP always play the most important role in the national government formation. After this move, a new positive charge will flow in congress cadre which was feeling low. Now, Congress may start playing on front foot. When two major regional outfits, the Samajwadi Party (SP) and the Bahujan Samaj Party (BSP), formed an alliance to take on Modi’s Bharatiya Janata Party (BJP), they left out the Congress from the “grand alliance.” India’s grand old party then seemed to be on the backfoot.

Priyanka is likely to contest from Raebareli, the traditional seat of Gandhi family, currently represented by Sonia Gandhi. Priyanka bears resemblance to Indira and is likely to attract female voters who will connect with her easily. Gandhi is no stranger to politics, she has been working from the background for a long time, though her public appearance was limited to Raebareli and Amethi. Last year during the Assembly polls, she was the star campaigner of the party and took part in numerous road shows and rallies. Just a couple of months ago, she was part of some closed-door meetings to decide the chief minister of Madhya Pradesh, Rajasthan, and Chhattisgarh.

The decision to appoint Priyanka Gandhi shows how Congress leadership had to give in to the demand of its workers and people. For the past several years, the workers have been demanding her presence in national politics. She has always been rated a more spontaneous politician than her brother. she is known for her ability to connect with the masses and also for her articulate in Hindi.

Rahul Gandhi said “I am very happy that my sister Priyanka will assist me in UP in the Lok Sabha polls. She is very capable”.

Will the move backfire?

There is no doubt that UP is still under the effect of the Modi wave. As whole India in UP also two rival parties come together and left Congress alone. Both BSP and SP will stand their candidate on 50% shares while Congress will stand their candidates on all over 80 seats. This may decrease the vote share of Congress, BSP, and SP and will directly benefit to BJP.

Be the First to comment.
New Features in Statuspal 2.8
By: Eduardo Messuti
Tags: Markdown, Twitter, Monitoring, Status Page, SaaS

We have been working hard to bring you new and highly requested features, this are the most remarkable in this release:

3rd Party Monitored Services

You are now able to configure your 3rd party monitors to automatically create incidents in your status page when their checks fail.

We provide you with an inbound email per service that you can easily setup as an alert contact email in your 3rd party monitoring check.

To find out more go to our knowledge base.

Twitter Handle & Contact Email

You can now configure a public twitter handle that will be displayed in the footer of your status page, as well as a contact email that will be displayed in the header as well as in the footer.

Status page header with support email contact

Status page footer with twitter & support links

Markdown Support

We now support markdown in the incident updates, so you can get creative and make them look just the way you want.

Maintenances start & end date displaying Improvements

We have improved and made more uniform the way in which we display your maintenance incidents, now they always display the start and end date, in the website as well as in the subscribers emails.
Encrypted Messaging Protects Your Data
By: Apollo Fintech
Tags: Apollo, Blockchain, Privacy, Monitoring

by Marvin Dumont

No one likes to be under 24/7 surveillance. The practice is creepy, and it may be illegal in your jurisdiction.

Yet it persists.

The Deep State likes to know everything all the time, and this clashes with peoples’ preference for privacy. Unfortunately, Facebook, Google and big telecom companies are known to share users’ info with third parties, including the government, without disclosing such practices.

Apollo Foundation believes that privacy is essential to individual liberty. Encrypted Messaging on Apollo blockchain allows any user

to send and receive untraceable messages, as well as, data files from one account to another.

Check out the Messenger feature on Apollo Wallet which is a secure channel for communication.

It can be risky to disclose sensitive info: Hackers work day and night to steal your property or identity. But you have options for protecting sensitive messages.

Encrypted Messaging utilizes Olympus Protocol. The user’s IP address and the transaction carrying the message are invisible.

Apollo (APL) all-in-one privacy currency combines features of mainstream cryptocurrencies into a single, unregulatable platform. With 2-second block speed, it’s also one of the fastest cryptos on Earth — faster than Ripple (XRP).

The platform lets you transact and send data in total anonymity via tech breakthroughs such as private ledger, decentralized exchange, coin mixing, and advanced IP masking.

IP Masking 2.0 uses encrypted-data protocol to hide your real-time physical location. It may be possible to use the platform in restrictive jurisdictions such as China where Tor is blocked. (See article.)

Coin Mixing (Apollo Mixer) is groundbreaking tool (built by our developer team) that hides the sender, recipient and transaction amounts. It obfuscates the money trail so you can do business safely and privately. (See article.)

Learn more about Apollo’s groundbreaking tech:

Apollo Mixer: A Cloaking Tool That Makes Your Money Untraceable

Apollo To Release Hermes Update By Q1 2019

IP Masking 2.0: It Keeps You Anonymous And Safe

Apollo: A Super Fast All-In-One Currency

Adaptive Forging Streamlines Apollo Blockchain

Apollo Pushing The Boundaries Of Blockchain Speed

Bitfi Wallet Integrates Apollo All-In-One Privacy Currency

Sharding To Speed Up Apollo Blockchain

Recent Articles:

Protect Your APL With Bitfi Wallet

CoinPayments Integrates Apollo: Now A Payment Option At Online Stores

Netcoins Listing: Apollo Available At 171,000 Retail Locations Globally

5 Benefits Of Total Anonymity

Apollo Breaks Top 100 In Market Cap

Apollo’s Financial Upside And Top-10 Ranking

4 Benefits Of Fast Transactions

Apollo Commits To Decentralized Organizational Structure

Cryptos See Adoption In High-Inflation Countries
IoT based Algae Spirulina Growth Monitoring System
By: Appleton Innovations
Tags: IoT, Control, Monitoring, Algae, Internet of Things

What is Spirulina Algae?

Spirulina is a biomass of blue-green algae that can be consumed by humans and animals. Spirulina is a type of blue-green algae that’s loaded with nutrition: protein, antioxidants, and numerous vitamins and minerals.

It is a simple organism that grows easily in warm water. Just like plants, blue-green algae can produce energy from sunlight via a process called photosynthesis.

Health Benefits of Spirulina Algae

Extremely High in Many Nutrients

Powerful Antioxidant and Anti-Inflammatory Properties

Anti-Cancer Properties

Growing Spirulina Algae

Spirulina is high economic value microalgae, which is also known as an important natural source of protein.

After harvest, it can be made into healthy food through a drying process.

However, because algae can absorb toxins found in the environment, some people opt to grow their own spirulina at home in safe and controlled conditions.

IoT for Growing Spirulina Algae

To produce the best Spirulina Algae, it has to be grown in an alkaline water media with a stable water temperature of around 86◦ F, controlled intensity of ultraviolet, regulated water turbidity, as well as maintained supply of Oxygen, Nitrogen, and CO2.

To be able to control temperature,turbidity and ultraviolet light, this project proposes an IoT based monitoring and automation system.

Project Objectives

Monitoring and controlling temperature,turbidity and ultraviolet light using IoT for growing Algae.

Project Aim

Internet of Things has become an inevitable choice of technology application for improving high quality food source, assuring food safety, and standardizing the forms of cultivated Spirulina Algae. Our Project uses a micro controller, such as Arduino, and sensors to create a real-time monitoring system via the internet to monitor water temperature, ultraviolet intensity, and water turbidity.

This project presents an intelligent IoT System, to monitor these data. We designed this project for data acquisition, data visualization, and data monitoring, based on the IoT concept

Components Required for building this project

NodeMCU

Turbidity Sensor

Temperature Sensor

UV Light sensor

Micro Water Pumps

Fan, Light

Driving Circuit

wires and other electronic components

Software used

Arduino

Thingspeak

IFTTT

Real time Implementation

Sensor Data Visualization

Implementation of monitoring and controlling system for Algae Growth can abundantly help improve the yield of the harvested algae.
Debugging CPU & Memory usage in Rails app
Tags: Memory Improvement, Monitoring, Ruby on Rails, Performance Tuning, Cpu

Photo by Isaac Smith on Unsplash

For the past few weeks, we were facing issue with the digital ocean machine hitting more than 90 % CPU limit for 5 mins continuously (We have set up monit to trigger a notification to our Telegram channel for it) and it is really annoying to see these many messages in telegram,

Monit alerts in the telegram

So in this post, I will log the approaches taken to debug the issue.

System configurations

Digital Ocean machine with 1 GB RAM, 25 GB Disk storage Debian box. Currently, everything is running here. Nginx for web server, Puma for app server (Rails), Postgresql database.

Figuring out the slow queries

Our first thought was to check how the database is performing. And we used pghero to see the number of queries executed and how long it is taking.

Pghero for analysing postgres queries

As you can see the topmost slow query is taking an average of 25 ms and it is the search query on profiles. According to our system specs, we felt that it is in acceptable limit.

Debug the memory usage

We want to take a look at memory usage because from htop we saw that most of the memory is taken and swap memory is also mostly full.

htop — mem & swap are mostly full

get_process_mem gem allows us to check the memory usage of particular piece of code and using GC object in ruby we can see how garbage collection is being done. Eg: you can see below which uses 641 MB of memory,

Profile memory

And you can see how much total objects are allocated and how much objects are freed after GC,

GC freed, allocated objects

We found the gem derailed_benchmarks which does all this for us. It can statically check how much is the memory used & objects allocated by the gems used in our project.

bundle exec derailed bundle:mem

We found that sprockets-rails gem occupies 2 MB, even though we are using the rails only for backend API. Apart from this, it is not of great use to us.

Object allocation per request

We integrated scoutapm to see which request allocates the most number of objects.
Sanity Testing an OpenShift Cluster
By: Guy Rakover
Tags: Openshift, Sanity Testing, Docker, Monitoring, Ansible Tower

Sanity Testing an OpenShift Cluster

Over the past few months I have had a few scenarios where customers with 3.11 Openshift clusters were looking for a way to run automated functionality and sanity tests to insure cluster health and functionality on top of the available Grafana/Prometheus monitoring. Now that I have delivered it I thought that it would be useful for others trying to tackle the same issues. Below are two particular scenarios:

Sanity Testing the Cluster for Basic Deployment Health and Functionality

Periodic Test for Insuring Access to External Resources and Deployment Functionality

OpenShift Cluster Sanity Tests

Initially designed to be run once a cluster is deployed using an Ansible Tower workflow, the below project is useful not only when deploying a new cluster but also when looking to run periodic sanity tests on the cluster. The procedure has 3 major stages: project creation, infrastructure tests and cluster reboot.

It covers the following scenarios:

Check project creation and pod deployment

Verify persistent storage is available

Router Connectivity

All Pods are running in the main OCP projects

Checks for failed pods in all projects in the cluster

Verify nodes are ready

Etcd health

Console health

Checks selinux policy is correct

Check nodes survive reboot

The project utilizes an nginx image and Ansible and can be found on github:

OpenShift Periodic Functionality Sanity Tests

I got another request regarding testing connectivity to external services from within the pod network as well as insuring that at any given time a new project could be created and a pod brought up on any compute node. The aim being to periodically check for any unexpected network connectivity issues and internal OCP resource issues. For this I put together the following project:

Based on Ansible and a custom container containing the s3cmd and psql CLIs as well as httpd, the project performs the following:

Creates a project

Creates a Route, Service, PVC and deployment config for the container

Runs through all the compute nodes 1 by 1, labeling and scaling up/down the deployment config so that the pod runs on each node

When the pod is up, it runs an internal script that connects to both S3 and Postgresql endpoint and writes the response to /var/www/html/results.html

Before moving the Pod to the next node, Ansible queries the file via the router and checks for errors in the results.html file

The final architecture for implementing the solution is utilizing Ansible Tower Job Template Scheduling and SPLUNK logging aggregator as follows:
DOCKER SWARM MONITORING. This post is related to the monitoring…
Tags: Restart, Containers, Docker, Monitoring, Docker Swarm

Universal Control Panel

This post is related to the monitoring capabilities provided by the UCP to monitor all the deployed services .

Things I do

By using UCP , We can also plan for deploying/updating the containers by the sections it provides to add the necessary variables to start the services .

Above Pictures shows on how we can update the service by adding the new docker image and adding all the necessary details fields listed on the left side and then saving the changes to take effect .

Restart , Rollback and Remove and scheduling actions are used on day to day activities when there’s an issue on containers getting hung or when the load is high and also we can choose when the container stops responding to the requests .

Manage Users and Permissions

This panel also has security authentications to control the access depending on the requirement and designations in a project . Only who has admin and deployer rights will be given the access to view the resources allocated for their project and also the role to perform certain actions required by the team members .

Secrets , Volumes , Networks and configs

Secrets and configs are used for saving the passwords and also the files required to be different as per the environments by giving the source and destination instructions in the compose file or in the UCP. The same way we can do for volumes and networks. I am using the above features mostly in the compose file as i didn’t get an opportunity to use on the UCP.
Use DaTa FraMe vaLues to fill HTML fOrMS via Selenium
By: Anum Sadozaeii
Tags: Selenium, HTML, Pandas Dataframe

This article is the next phase of my previous article. How to create users. In this phase, we will upload the data from a.csv file to the HTML forms.

Let’s start...

1- Upload CSV file

2- Create Data Frame

2- Use Selenium to fill HTML forms with data frame values

from selenium.webdriver.support.select import Select

import pandas as pd



try:

# Read CSV File

client_df = pd.read_csv('D:/.../generate_user_data.csv')

name = client_df.iloc[:, 0]



# Create Users

driver.implicitly_wait(30)

driver.get('your link')



# for loop

size = len(client_df.index)

for a in range(size):

driver.implicitly_wait(10) # Create new column with . between names client_df.iloc[:,0] = name.replace(' ', '.', regex=True) # Enter User Name from the Dataframe driver.find_element_by_xpath("// input[ @ id = 'Username']").send_keys(str(client_df.iloc[:,0].values[a]))



# Enter Email

driver.find_element_by_id('Email').send_keys(str(client_df.iloc[:,1].values[a]))



# Enter Password

client_df['password'] = "pwd"

driver.find_element_by_xpath("// input[ @ id = 'Password']").send_keys("pwd")

driver.find_element_by_id ('ConfirmPassword').send_keys("pwd")



# Select the Role

level = Select(driver.find_element_by_id('RoleName'))

level.select_by_visible_text('Admin')

driver.implicitly_wait(10)



driver.execute_script ("window.scrollTo(0,document.body.scrollHeight)")



# Click Create

driver.implicitly_wait (10)

driver.find_element_by_class_name('card-footer').submit()



# Go to the link again

driver.get ('your link')





finally:

driver.quit ()

In this code, I am creating user credentials.

Create new column with . between names

This means my application do not accepts white spaces. So i have added a dot. Enter User Name from the Dataframe

This means the data frame i have created with the .csv file, its values are fetched here in the web HTML form using selenium. You must have noticed that i am converting the value into string. As the textfield input is string so that's why am using str().

client_df.iloc[:,1] you can easily understand this thorugh my story i wrote about

This means the data frame i have created with the .csv file, its values are fetched here in the web HTML form using selenium. You must have noticed that i am converting the value into string. As the textfield input is string so that's why am using str().client_df.iloc[:,1] you can easily understand this thorugh my story i wrote about Pandas Data fRAME Cheat Sheet — select specific CoLumn and RoW

I believe the rest of the code is self-explanatory. if you have read my previous stories. You must know why I have not added the code chunk for calling browser, getting URL…..

Stay happy and Safe!
Hacking my Building’s Gym Reservation During the Pandemic
By: Adnan Haque
Tags: Selenium, AWS, Programming, Fitness, Problem Solving

Background

A bit about me — I work in the data science division of a company building predictive models for the insurance industry. Going into college, the combination of lack of activity and increased consumption of junk food lead to me gaining the “freshman fifteen.” Unfortunately, the weight gain didn’t stop there, and I continued to put on ~5–10 lbs per year for the decade that followed. Six months before the pandemic and with my 30’s around the corner, I decided enough was enough, and it was time to make a serious commitment to getting my health in order.

When the pandemic hit, I was 30 lbs towards my goal and resorted to at-home workouts to keep up my progress. I was having difficulty getting the same amount of strength training using makeshift dumbbells made from water jugs. So when my building decided to re-open their gym, it was welcome news.

I’m fortunate enough to live in a building that was proactive in establishing pandemic precautions without shutting down everything in the building. Their solution to having people in the gym potentially spreading the virus was creating gym “zones,” each of which could only have one person at a time and had to be booked in advance through a web portal. Only one zone had access to a squat rack and dumbbells. With a dozen spots per day and a few hundred people in my building, getting a spot was going to be difficult.

Solution

Reservations for the day can be booked a day in advance and open at midnight each day. The first day I logged in at 12:15 AM, and my active neighbors had already reserved every timeslot. Thinking I’d be prepared the next time, I set an alarm for 12:00 AM the following day. The portal froze when I tried logging in, I was finally able to load the page at 12:05 AM and every reservation was already booked except for the 6:00 AM time slot (I am not a morning person). With the combination of mild frustration and ample time due to the pandemic, I decided to turn to a programmatic solution to my problem. I wrote a quick script in Python using Selenium to navigate the web portal and book a reservation. After a bit of testing, I scheduled it to run on my desktop using Crontab.

The next morning I woke up expecting a 6:00 PM booking of the gym for the next day, but nothing was booked! When I looked at the logs, I noticed that while the script was triggered at midnight, the actual booking wasn’t attempted until fifteen seconds after midnight, at which point almost every timeslot had already been booked. People were either frantically refreshing the website at midnight or looking at creative ways to book the gym.

Having become a little obsessed, I made several improvements to the application over the next two days, each in response to a challenge I encountered. Below are a few updates:
How to explicitly check the page load in Selenium?
By: Vishal Gada
Tags: Selenium, Test Automation, JavaScript

While writing test automation scripts we know that sometimes web pages takes more than usual time to load due to various factors like- internet speed, Ajax calls, leading our script to fail. In this blog today, I would explain a way to explicitly check the page load of in Selenium.

We will use “Document.readyState” to determine the document status. This function has below values:

Loading- The document i.e the HTML page is still loading.

Interactive- The HTML page is loaded but the CSS and other resources are still being parsed.

Complete- The document is completely loaded along with all its sub resources.

To implement this in Selenium, we will use the ExpectedCondition interface to write a new condition, where we would wait till the value for “Document.readyState” is “Complete”. To avoid for the script going into infinite loop, we will specify a maximum timeout in the explicit wait to check for this condition. Below is the snippet of the code which can be used to check for page load:

public void waitforDomComplete(WebDriver driver) {

ExpectedCondition<Boolean> pageloadStatus = new ExpectedCondition<Boolean>() {

public Boolean apply(WebDriver driver) {

return ((JavascriptExecutor) driver).executeScript(“return document.readyState”).toString().equals(“complete”);

}

};

WebDriverWait wait=new WebDriverWait(driver,30);

wait.until(pageloadStatus);



}
Chrome as a service for Rails testing
By: Josef Šimánek
Tags: Selenium, Webdriver, Ruby, Docker, Chrome

Meanwhile working on next part of Ruby C extension series I have faced different problem.

Since it is more and more painful to install awesome capybara-webkit I have started to look around for some reasonable replacement. My requirements are really simple. It must be really easy to use it locally for developers and it should be possible to run it in containerized environment (Docker on CI in my case). Let’s generate example Rails 5.2 application we can use for testing based on my prepared template.

There’s is one system test created testing some javascript execution. By running our system test suite we can easily check if our solution works. Let’s try it!

If you have Chrome locally installed you should see new browser window for short time used to execute our system test. It works out of box! Nice! Our first requirement is met. There’s no need for any local configuration, no need to install any additional dependencies. All you need is Chrome installed. No more pain having proper Qt installed to compile capybara-webkit locally.

test app running

simple system test passing only when JavaScript is executed

Docker

Let’s slowly prepare application for containerization in Docker by adding simple Dockerfile using thin Alpine official Ruby images.

Pretty simple Dockerfile , isn’t it? We need to install tzdata package and temporarily install build-base and ruby-dev packages to be able to compile nokogiri. Let’s try to build and start our app.

App is running, we can reach it locally at http://localhost:3000/home/index. You should see “Hello from JS” text. Let’s try to run specs in our new container.

Since we are using really thin docker base image (based on Alpine linux) Selenium WebDriver can’t start Chrome because it is not available in image. We can fix it by installing it into the image, but it will almost double our image size and also it will probably lead into two dockerfiles, since we don’t need Chrome installed in production container.
Selenium WebDriver’ınıza Kulak Verin | Event Firing WebDriver
By: Emirhan Gül
Tags: Selenium, Testing, Test Automation, Selenium Webdriver, Eventfiringwebdriver

Benim implement edilen metodları kullanım şeklim aşağıdaki gibidir:

/*

* FINDING ELEMENTS | FINDELEMENT() & FINDELEMENTS()

*/

// Called before finding Element(s) @Override

public void beforeFindBy(By by, WebElement element, WebDriver driver) {

System.out.println("Trying to find: " + by.toString());

waitAllRequests();

} // Called after finding Element(s)

@Override

public void afterFindBy(By by, WebElement element, WebDriver driver) {

System.out.println("Found: " + by.toString() + "'.");

waitAllRequests();

} /*

* CLICK | CLICK()

*/

// Called before clicking an Element

@Override

public void beforeClickOn(WebElement element, WebDriver driver) {

System.out.println("Trying to click: '" + element + "'");

// Highlight Elements before clicking

for (int i = 0; i < 1; i++) {

JavascriptExecutor js = (JavascriptExecutor) driver;

js.executeScript(

"arguments[0].setAttribute('style', arguments[1]);",

element, "color: black; border: 3px solid black;");

}

waitAllRequests();

} // Called after clicking an Element

@Override

public void afterClickOn(WebElement element, WebDriver driver) {

System.out.println("Clicked Element with: '" + element + "'");

waitAllRequests();

} /*

* URL NAVIGATION | NAVIGATE() & GET()

*/

// Prints the URL before Navigating to specific URL @Override

public void beforeNavigateTo(String url, WebDriver driver) {

waitAllRequests();

} @Override

public void afterNavigateTo(String url, WebDriver driver) {

waitAllRequests();

} @Override

public void beforeNavigateBack(WebDriver driver) {

waitAllRequests();

} @Override

public void afterNavigateBack(WebDriver driver) {

waitAllRequests();

} /*

* ON EXCEPTION | SCREENSHOT, THROWING ERROR

*/

// Takes screenshot on any Exception thrown during test execution @Override

public void onException(Throwable throwable, WebDriver webdriver) {

System.out.println("Caught Exception");

File scrFile = ((TakesScreenshot) webdriver)

.getScreenshotAs(OutputType.FILE);

try {

FileUtils.copyFile(scrFile, new File(

"/Users/sahabt/Documents/MavenDev/gaugebasics/screenshots/"+System.currentTimeMillis()+".png"));

} catch (Exception e) {

System.out.println("Unable to Save");

}

}

Umarım sizler için faydalı olmuştur! Bir sonraki yazımda görüşmek üzere, mutlu testler 🤘🏼

Referanslar:

https://www.seleniumeasy.com/selenium-tutorials/webdriver-event-listener-example

https://www.softwaretestingmaterial.com/webdriver-event-listeners/ https://www.vskills.in/certification/tutorial/selenium/event-firing/ https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/support/events/EventFiringWebDriver.html http://www.testingdocs.com/eventfiringwebdriver/