fabiochiu/medium-articles huggingface dataset:
ON THE NEW ‘SIMILAR TERMS’ FEATURE IN SHARE-OF-SEARCHING
By: Frank Harrison
Tags: Search, Marketing, Share Of Search, Search Trends, Data Science

5 July 2021|Brands, Data Science, Insight, Search, Search trends, Share of Search

When you make a Google search, Google’s algorithms display ‘Related Searches’ at the bottom of the results page and a list of ‘People Also Search For’ terms that may appear on the right side of the page.

‘Related Searches’ are search terms related to the term that was searched for. They provide useful insight into the kind of information people are looking for related to a search; the questions they ask, and their interests and intent. However, there is no indication of the relative search volume of the related terms to each other or to the original search that was made. Nor is there any indication of the trend in search interest for the terms.

The ‘People Also Search For’ list shows Google-determined topics or products that are connected to the original search — another way that Google provides related queries that reflect the wider intent of the user. As with ‘Related Searches’, there is no indication of relative search volumes or trend in search interest for the list.

This week Croft Analytics has added a ‘Similar Terms’ feature to its Share-Of-Searching tool that shows both the relative level of search volume and trend over time for ‘Related Searches’ and the ‘People Also Search For’ list. Users can now enter a single search term to see the trend (back to 2004) in relative search volumes and share of search for the entered search term alongside ‘Related Searches’ terms and the ‘People Also Search For’ list. The results can also be filtered by country and date period.

Here is an example of an output from Similar Terms showing relative search volumes for a search term — in this case ‘Tesla’ — along with its ‘Related Searches’ and ‘People Also Search For’ terms.

The chart shows a 3X rise of global search interest in Tesla between January 2015 and June 2021, with big spikes in interest around the times of major Tesla launches and events. The chart also shows that the two largest similar terms to Tesla are Porsche and Lexus. There was also a large spike in interest for SpaceX in May 2020 (astronaut launch). Several of the similar terms have also become increasing popular in the last two yearsincluding NIO (Chinese electric vehicles), Tesla stock, Tesla price, and Tesla truck.

To see relative search volumes and trends in search interest for any search term, take a look at the ‘Similar Terms’ feature at https://www.shareofsearching.com

— — — — — — — — — — — — — — — — — — — — — — —

By Frank Harrison, Croft Analytics
Implement a search and filter in flutter
By: Ezaldeen Sahb
Tags: Search, Flutter Widget, Flutter, Filters, Flutter Mobile App

There are many flutter widgets to implement search feature just like in Android, yet there is a simpler way to implement this using flutter SearchDelegate class which gives you almost all that you want in only 4–5 simple steps so let’s see what we can do with this

Flutter search

We will need to explore two classes before we start

When we want a dynamic toolbar that when we slide or expand it shows us content, we use the great widget called SliverAppBar which always be the first child for CustomScrollView.

This will make a cute app bar with two icons

Next we will implement the search feature.

CustomSearchDelegate

To make a custom search , you need to create a new class that extends the SearchDelegate class. after overriding the SearchDelegate method you will have this class

Now let's build our search class, a simple class will be shown below

Now we just have to connect our cute search app bar with our custom search class

if you remember our search button icon from the above silver app bar we just need to add the showSearch method to connect the app to the searchClass, the rest is magic ^-^.

the search widget view

I added some optional features like instant search suggestions and recent search suggestions to the example, the source code can be found here.

https://github.com/Ezaldeen99/search_demo
How searching on mobile browsers is different from searching on desktop browsers?
By: Searchblox Team
Tags: Search, Elasticsearch, Mobile, Smartphones

In today’s world, smart phones and tablets have completely dominated everyone’s life — yours and mine included. It’s no surprise that mobile search has surpassed desktop search. Consumers are searching for answers from their mobile devices. Research shows that more than 57% of online traffic comes from mobile devices.

With so much on the line, Here are a few differences you might have overlooked about search on mobile browsers vs search on desktop browsers.

Mobiles have a smaller display

Mobiles have a smaller display than desktops. So the user will not be able to view all the information at one go. Unless the required information is clearly visible, you shouldn’t expect the user to stick around. So the page layout for a mobile device needs to be simpler, easily readable and in general optimized for touch devices.

Quicker results on mobiles

A good thumb rule to follow when it comes to building applications specific to mobiles, is to make things super fast because the chances that the user is on the go is very high and you don’t want to lose that user because it took a second longer to load.

Mobile devices are easy to use

Smartphones have become very ubiquitous, infiltrating every part of our lives. This is mainly because it brought the power of technology to our palms and that ease with with everything happens is the unsung hero here. There are millions of apps out there overall, and thens of thousands for each and every category. So your user is spoilt for choices, you don’t want to give them a reason to find some app that’s not yours because yours was harder to use.

Mobile devices may have slower processors

Not everyone who owns a smartphone is using the top of the line smartphone and this means the majority don’t have the super fast processors. Now it becomes your responsibility to build optimized apps that work on processors no matter its speed.

We at SearchBlox have been working on enterprise search for the past decade and a half and these were some of our observations in that period. This is why our products support mobile browsers right out of the box and don’t cause any overloads.

If your’e interested, we could set you up with a demo and you can judge for yourself how much good it would do to you. All you have to do is give us a call at +1 (866) 933–3626 or fill out this form.
Introducing TellusR for Solr search enhancements
By: Hallvard Ystad
Tags: Solr, Tellusr, Search

Part I − Implementing features that Solr lacks

According to a report from Forrester Research, 43 % of all visits in webshops start by using the search. I have previously discussed how webshops are improving their search functions, but there still is a gap between the performance of site internal searches in e-commerce and that of search engines that are not part of a webshop.

In Sannsyn, we work a lot with this. We have customers in the public sector and in web commerce, retailers and wholesale businesses, and they all need help with their search. Most of our assignments concern Apache Solr, which is one of the most popular search engines in the world. Solr is extremely fast and highly customizable. In fact, its very high number of customizable features and its enormous amount of toggleable buttons and levers are in themselves part of the reason people need help.

The Solr dashboard with various technical details

Solr has a visual dashboard where users can inspect various parts of the search engine: consult key figures, try out searches, read error descriptions etc. But all of this is for the IT technicians, who check to see that the installation is doing all right, that it is not using too much of the server resources, etc. The webshop manager, on the other hand, would rather know what people search for this month, which searches result in the highest share of convertions, how many searches simply fail, etc. But this is not visualized through the Solr web interface.

TellusR to the rescue

So we set out to create a dashboard that could make Solr useful for both techies and webshop managers: show graphs for conversions per search, alert about searches that yield zero hits, give information about what search terms are trending etc. We also wanted to enable non-tech staff to add synonyms to a Solr search or to elevate certain search terms or use some of the other features of Solr that are otherwise only available to those who know the Solr vocabulary and master the art of XML editing.

The result is TellusR. We call it a Solr Refiner.

The TellusR Solr Refiner consists of three parts. There is a graphical user interface (TellusR/Central); a component doing natural language processing (TellusR/NLP); and finally a Solr integrator, which brings information back and forth between your Solr installation and the two other components.
Deploy Appbase.io On Heroku With One Click
By: Siddharth Kothari
Tags: Search, Elasticsearch, Appbase, Heroku, Search Engines

This post walks through the process of deploying an appbase.io instance with Heroku with a one click deploy button.

Click/tap to deploy appbase.io on Heroku

Appbase.io supercharges the Elasticsearch experience by offering:

Out-of-the-box search and click analytics Read More,

Search relevancy APIs and a control plane to visually create relevant search experiences Read More,

True access control and security for search Read More

Besides these, it also offers an improved developer experience by allowing to import your data via the dashboard GUI / CLI / Zapier and provides the ability to build a relevant search UI with zero lines of code.

The following video shows a demo of how appbase.io can be deployed with Heroku with the one-click deployment option.

Environment Variables to Configure

Once you deploy with the above button, you will see the Heroku configuration screen. You will need to configure the following variables over there:

APPBASE_ID — You will receive this ID after registering the instance by visiting https://dash.appbase.io/install URL

— You will receive this ID after registering the instance by visiting https://dash.appbase.io/install URL ES_CLUSTER_URL — This is the URL of your Elasticsearch instance, e.g. AWS Elasticsearch, Elastic Cloud, self-hosted Elasticsearch. If your Elasticsearch is secured with Basic Auth, you can enter the basic auth credentials in the URL string in the format of https://${username}:${password}@es-host:port

— This is the URL of your Elasticsearch instance, e.g. AWS Elasticsearch, Elastic Cloud, self-hosted Elasticsearch. If your Elasticsearch is secured with Basic Auth, you can enter the basic auth credentials in the URL string in the format of USERNAME — This is a root username you can choose for accessing appbase.io either via API or via the dashboard at https://dash.appbase.io

— This is a root username you can choose for accessing appbase.io either via API or via the dashboard at PASSWORD — This is the root password you can choose for accessing appbase.io either via the REST API or via the dashboard.

Note: The root username and password provide you an admin access and aren’t recommended to be used for a live search application. You can use these in the dashboard to create credentials with narrower permissions.

That’s all! Your appbase.io instance would be running on Heroku in less than a minute after configuring the above.
Release Wrap Up — August 2020. Search, Dashboard, Hub, Oh My!
By: Alex C-G
Tags: Search, Neuralsearch, Software, Opensource, View full release notes

It’s been a busy month here at Jina — our team has been beavering away on our core product, Jina Search, and our other projects like Jina Dashboard and Jina Hub, both of which are getting their first releases this month. They grow up so fast!

So, needless to say, we’re excited to share the highlights of our release notes.

We’re just sharing a smattering of highlights here. Be sure to click through for the full notes!

Jina Search 0.5.0

Jina Search provides the core of our neural search framework.

Recursive Document structure

Native data querying capabilities

Migration of Executors to Jina Hub

Support for Mindspore.

Jina Dashboard 0.1.0

Get insight on running tasks in Jina. With Dashboard, you can analyze logs, design Flows and view Jina Hub images.

View log stream. When running a Jina instance with log server enabled, you can view logs output by your instance in the log stream tab by connecting it to the Dashboard in the settings tab.

tab by connecting it to the Dashboard in the tab. Monitor Tasks. You can view network load, queries per second, and messages sent and received by your Jina instance in the task tab.

tab. View and Edit Flows. In the flow design tab, you can view the Flow of a running instance, import a Flow in YAML format, edit and export a Flow in YAML format.

tab, you can view the Flow of a running instance, import a Flow in YAML format, edit and export a Flow in YAML format. Explore the Hub. In the hub tab, you can view available images, and explore more information about each one such as setup instructions, README's, and build history.

Jina Hub 0.1.0

A centralized registry of custom and community-driven Jina Apps and Pods that are tailored to specific use cases of neural search.
How to store Sample Result Data in JMeter Load Test?
By: Bharat Sinha
Tags: Load Testing, Jmeter, Monitoring, Data Science, Analytics

This article aims to answer the question asked in the title of the article but suggesting only a particular solution to the problem.

JMeter is one of the most popular tools in the software industry for load testing products and services. But a test is useless if you do not have the statistics and data of the entire test. As JMeter is a mature enough product it allows us multiple ways to achieve that. Here we are going to discuss just one way which I found useful because of the complexity and specific requirement of our product.

JMeter provides BackendListener which basically exposes the SampleResults object to a class running in a separate thread. By default, JMeter comes with influx DB implementation of the BackendListener but if you are not sure about influx DB and need flexibility in terms of where your data needs to be sent and stored you can use a generic version of the BackendListener written by me which now I have made open source.

It is a simple implementation which simply sends data to any POST based API endpoint specified in the configuration.

Some of the features are:

Support any POST based endpoint.

based endpoint. Leave data management to the backend making the implementation lightweight.

Uses switchable GZIP compression to decrease the packet size sent to the endpoint.

compression to decrease the packet size sent to the endpoint. Simple plug and play allow quick turn around time.

Control the frequency of data being sent to your endpoint.

A SampleResult object sent from API looks something like this:

{

ResponseCode: '200',

ContentType: 'text/html; charset=utf-8',

EndTime: 1566502823484,

IdleTime: 0,

StartTime: 1566502823292,

ResponseMessage: 'OK',

ErrorCount: 0,

Assertions: [],

Success: true,

URL: 'https://github.com/',

Latency: 151,

Bytes: 90265,

AllThreads: 1,

BodySize: 87922,

ThreadName: 'Thread Group 1-2',

DataType: 'text',

ResponseTime: 192,

SampleCount: 1,

ConnectTime: 58,

SampleLabel: 'HTTP Request',

timestamp: 1566502823292

}

The jar file for the BackendListener can be downloaded from the Maven repo. This supports various package/build manager i.e. Maven, Gradle, etc to support your project requirement.

This is the first release and I am looking for ideas and useful features, so, feel free to visit the repo and open your pull requests or simply raise an issue requesting a feature. Also, feel free to comment on your views and suggestions to improve this article.

PS: There are many ways to achieve the monitoring of JMeters results and this article in no way claim this method to be better or worse than any other methods. It simply suggests one-way doing things if it fits your requirement.
Have we lost our privacy?
By: Mohammed Moey
Tags: Social Media, Loss Of Privacy, Monitoring

I always find myself mindlessly opening Facebook and Instagram throughout my day for a cheeky scroll, updates on sports news, or laughing at memes to lighten up the mood after a long and stressful day at work. I use digital technology and socials on a consistent and regular basis and believe in a sense, social media does provide the world with access to things without having to search for them. Looking at Gen alpha kids born within the last 10 years, I see dependence as well as an increase in reliance on their use of technology. And I see it only going up from here with new technology releases controlling our future life and successes.

With the increased use of digital platforms, devices, and technology, the development of monitoring and surveillance increases. Have you ever felt like someone was watching your every step on a quiet winter night on your way home? You start to feel paranoid, scared, and agitated. So why do we not care that our every step online is being monitored by simply logging in?

Now it is concerning reading The Watchful Eye of Social Media report, presenting figures such as 89% or 3 billion social media users worldwide are actually being monitored. Does this really stop us from using social media? Younger people are less phased with their data and info being collected and monitored (Zurowski, 2007). Similarly, the more we keep relying on the internet with the thought we cannot live without it, the more comfortable we become with our personal data being collected.

I understand the fact passive monitoring is put in place to keep me and others safe with the removal of hatred and violent activity online. This is because us humans change our behavior and how we go about things when we know we are being watched by the big brother (Jacobs,2009, p.19). One of the biggest reasons why spying takes place by internet giants such as Facebook, is the capture of who we are, what we search for, click and view. This data is then sold to advertisers to linger ads in front of us. This is a great business for them but one in two individuals surveyed admitted they are uncomfortable with being tracked that way.

What does this really mean in terms of controlling our privacy?

Using social media sites such as Facebook has its consequences. As Facebook argued in court, “there is no invasion of privacy at all, because there is no privacy”, come to think about it though, you would think the details you provide Facebook is kept safe and secure without being shared with third-party companies but it is the complete opposite. The fact is, we don’t really know who knows what about us and Facebook does not want to be transparent with what details they share with companies. This to me feels like my privacy of using my personal social media has been taken away affecting my future experiences and use of social media.
SigNoz : Open-source alternative to DataDog
By: Pranay Prateek
Tags: Monitoring, Microservices, Application Monitoring, Prometheus, Observability

Introducing an open-source alternative to DataDog for privacy & security conscious companies who are concerned about HUGE saas bills — signoz.io

Here’s our GitHub repo

More and more companies are now shifting to a cloud-native & microservices based architecture. Having an application monitoring tool is critical in this world because you can’t just log into a machine and figure out what’s going wrong.

We have spent the last couple of years learning about application monitoring & observability. What are the key features an observability tool should have to enable fast resolution of issues.

In our opinion, a good observability tools should have

Out of the box application metrics

Way to go from metrics to traces to find why some issues are happening

Seamless flow between metrics, traces & logs — the three pillars of observability

Filtering of traces based on different tag and filters

Ability to set dynamic thresholds for alerts

Transparency in pricing

User experience not great in current open-source tools

We found that though there are open-source tools like Prometheus & Jaeger, they don’t provide great user experience like SaaS products do. It takes lots of time and effort to get them working, figuring out the long term storage, etc. And if you want metrics and traces, it’s not possible as Prometheus metrics & Jaeger traces have different formats.

SaaS tools like DataDog and NewRelic do a much better job at many of these aspects

They are easy to setup & get started

Provide out-of-box application metrics

Provides correlation between metrics & traces

But it has the following issues
Collectd介紹及安裝
Tags: Hardware, Collectd, Monitoring, Visualization, Monitor

1. 介紹

collectd從各種來源收集指標，例如操作系統，應用程序，日誌文件和外部設備，並存儲此信息或通過網絡提供。這些統計數據可用於監控系統，發現性能瓶頸（即性能分析）和預測未來系統負載（即容量規劃）。或者，如果您只想要私人服務器的漂亮圖表並且厭倦了一些本土解決方案，那麼您也可以在正確的位置;）。

一個圖表可以說超過一千個單詞，所以這裡是一個圖表，顯示了過去60分鐘內系統的 CPU利用率：

還有其他類似於collectd的免費開源項目 — 相關網站頁面上列出了一些鏈接 。那麼你為什麼要使用 collectd？還有，我們認為設定一些關鍵的差別 collectd分開。例如，它是用C語言編寫的，用於提高性能和可移植性，允許它在沒有腳本語言或cron守護程序的系統上運行，例如嵌入式系統。例如，collectd在OpenWrt上很受歡迎，家用路由器的Linux發行版。同時，它還包括處理數十萬個指標的優化和功能。該守護進程包含 100多個插件，從標準案例到非常專業和高級的主題。它提供強大的網絡功能，並且可以通過多種方式進行擴展。最後但同樣重要的是：collectd得到了積極的開發和支持，並且有詳細記錄。

2.安裝

解决依赖

rpm -ivh "http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm" yum -y install libcurl libcurl-devel rrdtool rrdtool-devel perl-rrdtool rrdtool-prel libgcrypt-devel gcc make gcc-c++ liboping liboping-devel perl-CPAN net-snmp net-snmp-devel wget https://storage.googleapis.com/collectd-tarballs/collectd-5.7.2.tar.bz2 tar xf collectd-5.7.2.tar.bz2

cd collectd-5.7.2

./configure #编译 make all install

或者直接执行命令进行安装

sudo apt-get install collectd

配置文件目录：/etc/collectd/collectd.conf

启动文件目录：/etc/init.d/collectd

日志文件目录：/var/log/syslog

数据存储目录：/var/lib/collectd/rrd/

若监控集群，则每台服务器都需要下载安装collectd并进行相应的配置。

3. 配置

sudo vim /opt/collectd/etc/collectd.conf

需要监控系统什么项就开启相应的插件，一般有cpu,memory,processes,load,interface,disk,swap等。

一般只需修改network插件这一项：

网络插件可以将数据发送到collectd的远程实例中(例如：数据库，csv文件或缓存等介质中)，或者接收从远程服务端发来的数据。

LoadPlugin network #去掉#就表示载入该插件

<Plugin network>

# # client setup:

Server "10.24.106.1" "25826" #该地址和端口是接收数据的服务器的地址和端口，例如:安装数据库（influxdb等）的服务器的地址和端口

# <Server "239.192.74.66" "25826"> #若需传输加密数据，就配置这一小块部分

# SecurityLevel Encrypt

# Username "user"

# Password "secret"

# Interface "eth0"

# ResolveInterval 14400

# </Server>

# TimeToLive 128

#

# # server setup:

# Listen "ff18::efc0:4a42" "25826" #如配置这一部分，表示接收从其他collectd实例中发来的数据

# <Listen "239.192.74.66" "25826">

# SecurityLevel Sign

# AuthFile "/etc/collectd/passwd"

# Interface "eth0"

# </Listen>

# MaxPacketSize 1452

#

# # proxy setup (client and server as above):

# Forward true

#

# # statistics about the network plugin itself

# ReportStats false

#

# # "garbage collection"

# CacheFlush 1800

</Plugin>

。

启动 执行命令 sudo /etc/init.d/collectd start

或者 systemctl start collectd

并设置开机启动 systemctl enable collectd

或者 systemctl start collectd 并设置开机启动 systemctl enable collectd 查看数据 若开启了rrdtool插件就可在/var/lib/collectd/rrd/目录下看到相应的统计数据。

通过下面命令可以查看具体数据：

rrdtool fetch *.rrd AVERAGE

*.rrd表示任何以.rrd结尾的文件，rrdtool命令更详细的用法可以自行百度

可以看到第一列为timestamp(时间戳)，可以通过命令date -d @timestamp把timestamp（以秒为单位）转换为和date命令显示的相同的的时间格式,命令date +%s用时间戳的形式表示时间。

./collectd -C ./collectd.conf 啟動

前提 要把設定檔跟執行檔 搬來安裝目錄 設定檔在 /opt/collectd/etc/collectd.conf 執行檔 在安裝目錄下 /tmp/program/collectd/collectd-5.7.2/src/deamon/collectd

關閉

[root@centoselk6 collectd-5.7.2]# ps -ef |grep collectd

root 27644 1 0 15:53 ? 00:00:00 ./collectd -C ./collectd.conf

root 28663 28475 0 16:10 pts/0 00:00:00 grep --color=auto collectd

[root@centoselk6 collectd-5.7.2]#

[root@centoselk6 collectd-5.7.2]#

[root@centoselk6 collectd-5.7.2]#

[root@centoselk6 collectd-5.7.2]#

[root@centoselk6 collectd-5.7.2]# kill -9 28663

利用logstash寫入es
Deploy Grafana 7 on OpenShift
By: Rafal Szypulka
Tags: Openshift, Grafana, Ibm Garage, Monitoring, Prometheus

Recently released Grafana 7 includes many new powerful features and UX enhancements like the unified data model, support for Jaeger data source, new data transformations, new visualization panels, and many more. In this blog, I will describe how to deploy Grafana 7 on RedHat OpenShift 4.x using the helm chart provided by Bitnami. The procedure includes also configuration of connection to the OpenShift platform Prometheus and migration of default Prometheus dashboards for OpenShift to our new Grafana 7 instance.

As a result, you will receive fully customizable and persistent Grafana 7 instances, where you can create or customize dashboards, configure users, teams, and organizations, as well as install new panels and data source plugins.

Photo by Benjamin Child on Unsplash

Prerequisites

Deployment Procedure

Prepare values.yml file with parameters that will be passed to the Grafana helm chart. Use the one included below as a starting point. It includes a list of ConfigMaps with default Prometheus dashboards for the OpenShift platform and the name of the secret containing data source connection to the OpenShift Prometheus. Specify persistance.storageClass or leave it commented out if you want to use your default storageclass .

Copy the following script to the directory where you have your values.yml file. Customize the GRAFANA_NAMESPACE variable if needed.

The script does the following actions:

Create the namespace for the Grafana 7 deployment.

Copy ConfigMaps with default Prometheus dashboards for OpenShift platform from openshift-monitoring namespace to your new namespace.

namespace to your new namespace. Create a service account ocp-prometheus and assign cluster-role view to the ocp-prometheus .

and assign cluster-role to the . Collect login token for the ocp-prometheus service account and use it for the definition of a data source connection.

service account and use it for the definition of a data source connection. Create a secret with a data source connection.

Add Bitnami repository to your helm client.

Deploy helm chart with customized input parameters provided with values.yml file.

file. Create a route for your new Grafana 7 instance.

Print Grafana URL.

Run the script in the local directory:

Wait until the script completes and verify that Grafana pods started successfully.

Collect the Grafana admin password:

oc get secret grafana7-admin --namespace grafana \

-o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode

Access the Grafana URL listed at the end of script output and verify the dashboards were successfully configured and live data is visible.
Kafka consumers health check in Spring Boot Actuator
By: Mateusz Jadczyk
Tags: Technology, Spring Boot, Kafka, Monitoring, Kafka Streams

Kafka consumers HealthIndicator

Thanks to Spring’s KafkaListenerEndpointRegistry we get access to a Collection of MessageListenerContainer which contains all the details regarding Consumers running in the system. You can iterate this Collection and transform it into a Map of chosen consumer properties.

When implementing a doHealthCheck(Health.Builder builder) method, you can use builder.withDetails(Map) to include status and details of each consumer separately and builder.status(Status) for the overall status of all your Kafka consumers.

Here’s an example of an implementation of the main method, mind that is uses some helper classes to hold the information in a readable way:

As we configured consumers to enter an Error state when they encounter a fatal exception, we’re just checking isRunning flag to compute a status of a particular consumer:

messageListenerContainer.isRunning() ? Status.UP : Status.DOWN

We’ve also included some extra information such as topic-partitions handled by the consumer which give you some insights into how the system operates.

The resulting response for a consumer health check looks like this:

Kafka Streams HealthIndicator

We define all Kafka Streams processors using StreamsBuilderFactoryBean . This allows us to autowire a List of all stream builders into the health check service and retrieve information. Again, the complete logic of digging out the necessary data is omitted, as everyone has their own needs.

Application ID can be fetched in a hacky way directly from StreamsBuilderFactoryBean :

streamsBuilderFactoryBean.getStreamsConfiguration().get("application.id");

For other details, look into KafkaStreams instance itself: streamsBuilderFactoryBean.getKafkaStreams() . Compute the status e.g. based on kafkaStreams.state().isRunningOrRebalancing() and by checking each thread separately:

You can also investigate thread details using kafkaStreams.localThreadsMetadata() to find out which tasks it handles and in turn which topic-partitions.

We found the following structure satisfying:
Why Use Monitoring Devices Applied to Transmission and Distribution Lines and Towers?
Tags: Monitoring

Why Use Monitoring Devices Applied to Transmission and Distribution Lines and Towers? Kimi Apr 13·2 min read

The application scheme of the intelligent patrol device in power line bird staying monitoring consists of image monitoring device, sound and light alarm, cloud platform, web browser, and mobile terminal. The image monitoring device continuously takes pictures and integrates The captured pictures are sent to the platform, and the platform intelligently analyzes the scene pictures.

When it finds that there are birds living or nesting on the site, it immediately issues an order to the device. After receiving the order, the device turns on the sound and light alarm to drive away the resident Birds: When the platform recognizes the bird’s nest, it will simultaneously push the information to the management staff through WeChat and SMS, so that the management staff can clean up the nesting objects on the tower in time to avoid line failure.

Solution Value

Continuously monitor the status of birds staying on lines or towers. When birds are found, they will send out alarm sound and light to drive away birds in time to ensure safe and stable operation of lines.

2. When a bird’s nest is found, the management personnel will be notified as soon as possible, so that the management personnel can clean up the nesting objects on the tower in time to avoid line failure.

3. SMS and WeChat receive real-time push information, and understand the status of birds in the line or tower anytime and anywhere.



If you need more information, please visit: IoT News, hope you have a good day.
Whats New on Statuspal 2.15
By: Eduardo Messuti
Tags: Web Development, Startup, Monitoring, SaaS, Slack

It’s been some months since we last wrote about Statuspal new features, but that’s not because we haven’t been hard at work, quite the opposite, we have delivered plenty of new features that our customers are already loving, here is a summary of the most important ones:

Localization & Multi-language Support

You are now able to configure in which language to display your status page easily, you can pick from one of our supported languages or configure a translation for a language we don’t yet support in a few minutes.

Supported languages panel in Admin site

You can configure your status page to support multiple languages, and publish your incidents with their respective translations.

To learn more about localization & multi-language support check out or help article.

Slack Subscriptions

Let your customers subscribe to receive Slack notifications from your status updates, it only takes marking a checkbox in your settings page to activate it.

Preview of a Slack notification from our bot

Learn more about Slack subscriptions in or help portal.

Annual Plans

You can now have your Statuspal subscription on an annual cycle, saving 10% compared to the monthly subscriptions, if you are on a monthly plan you can easily switch to an annual one on your status page’s plan page in your admin panel.

Annual plans now available!

Automatic Recurring Maintenances

It’s common to have scheduled maintenances that are performed on a recurring basis, like on Mondays every two weeks, or every 1st of the month, Statuspal lets you configure a scheduled maintenance to be automatically duplicated on a customizable schedule.

Recurring maintenance task configuration

Find out more about how to configure a recurring scheduled maintenance in this article.

Monitored Service Alerts Recipients

It is now possible to configure who in your team will receive the uptime alerts on your monitored services, in case the default email of your account is not the best one for it.

Monitored service configuration with configurable recipients

Added Support for <head> Code

It is now possible to configure custom HTML code to be inserted right inside the <head>…</head> code if your status page, this means you can highly customize your status page with custom styles and/or custom javascript, like for example adding your custom Google Analytics code, or an Intercom widget code.
AppDynamics vs Dynatrace — A genuine comparison (Part 1)
By: Abilashini Thiyagarajah
Tags: Monitoring, Application Performance, Appdynamics, Wso2, Dynatrace

Application Performance Monitoring(APM) is one of the hot topics in the current industry where we are depending on software applications to fulfil almost our every need. When we talk about software applications, the main consideration will be the performance. We should be aware of every possible reason which influences the performance of a software application. For that, we should monitor the application. That's where APM comes into the story.

When we decide to monitor our application, the next question comes into our mind is,

How are we going to monitor?

There are so many tools in the market which can be used for this purpose. Depending on our requirements, we can decide on the tool to be used. Here I am going to make your job easy by comparing two popular commercial APM tools against some important aspects.

First of all, let me introduce the two tools that I have chosen for my comparison.

Thankfully both the above tools have 15 days of a free trial. So I was able to give it a try and see how it works and compare depending on the results I got.

This article will be published in 3 parts. The first two parts will talk about the Architecture, Features, Deployability and Usability of the products separately. In part 3 we will see how we can deploy and monitor WSO2 Identity Server using these two products and compare the results.
Collect, Save and Report Linux System Activity Information
By: Gaurav Kar
Tags: Report, Performance, Linux, Monitoring

Photo by Austin Distel on Unsplash

SAR

The SAR command reports content of selected activity counters in the Linux System which are collected using sysstat over a period of time. You can select information about specific system activities using flags.The data can be redirected to an outfile which can be interrogated later.

Are you using a large SQL query and at the same time running your backup? Well we can report on the performance using the below commands.

CPU & Disk Performance:

We can run the below command to report on our CPU utilisation every 10 minutes.

[root@server ~]# sar -u

01:30:01 PM CPU %user %nice %system %iowait %steal %idle

01:40:01 PM all 0.11 0.00 0.09 0.00 0.02 99.78

01:50:02 PM all 0.10 0.00 0.09 0.00 0.02 99.79

Average: all 0.11 0.00 0.09 0.00 0.02 99.78 01:51:22 PM LINUX RESTART 02:00:01 PM CPU %user %nice %system %iowait %steal %idle

02:10:01 PM all 0.12 0.00 0.10 0.00 0.02 99.77

02:20:01 PM all 0.11 0.00 0.09 0.00 0.02 99.78

02:30:01 PM all 0.12 0.00 0.11 0.04 0.02 99.72

02:40:01 PM all 0.12 0.00 0.09 0.00 0.02 99.77



Average: all 0.12 0.00 0.09 0.01 0.02 99.76

We can also use the -b flag to report on disk utilisation.

Report on Network Activity

Do you want to see statistics from your network devices? For e.g. lets report data on our network interface cards. I only have one NIC on my Centos 7.

[root@server~]# sar -n DEV

01:30:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s

01:40:01 PM ens5 1.96 2.00 0.46 0.35 0.00 0.00

01:40:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00

01:50:02 PM ens5 1.93 1.96 0.45 0.35 0.00 0.00

How about printing data for a specific time period?

Lets filter the output based on timestamp and also pull up information from a particular log file by adding the file name as an argument to the sar command.
Exploring Sysstat Tools
By: Gaurav Kar
Tags: Performance, Linux, Monitoring

Photo by Michael Dziedzic on Unsplash

IOSTAT

Lets look at the disk activity on Linux. We can start seeing the speed of the reads and the writes to the disk which can be very handy to capture I/O metrics. The data can be viewed in megabytes/sec using the option -m or -k for kilobytes/sec.

[root@server~]# iostat -m

avg-cpu: %user %nice %system %iowait %steal %idle

0.21 0.00 0.17 0.22 0.02 99.38 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn

nvme1n1 0.06 0.00 0.00 14 2

nvme2n1 0.04 0.00 0.00 8 0

nvme0n1 2.43 0.06 0.01 479 66

PIDSTAT

This is very useful to report statistics on linux tasks. You can add in an interval which specifies the amount of time (in seconds) between each report and a count value in conjunction with the interval value to determine the total number of reports generated. The $$ signifies the current process id.

[root@server~]# pidstat -p $$ 5 3

02:24:53 PM UID PID %usr %system %guest %CPU CPU Command

02:24:58 PM 0 2484 0.00 0.00 0.00 0.00 1 bash

02:25:03 PM 0 2484 0.00 0.00 0.00 0.00 1 bash

02:25:08 PM 0 2484 0.00 0.00 0.00 0.00 1 bash

Average: 0 2484 0.00 0.00 0.00 0.00 - bash

MPSTAT

Another very useful command to use is mpstat to report on our processor related statistics. It lists each task that the CPU has to perform. The command can list statistics for all or just a specific processor ID. Use lscpu to display the processors and additional information on the system. In my example, I have 2 processors with id 0 and 1.
Working with Prometheus and Grafana — An Introduction
By: Akshaya Balaji
Tags: Grafana, Monitoring, Prometheus, Kubernetes, DevOps

Source: Google Images (Prometheus, Grafana, Kubernetes)

When we deploy software or application(s) in different environments, we want to monitor the environments, the performance of the deployed software or application(s), and other parameters. Monitoring is an important component of any technical field. There are many methods of monitoring like logs, metrics, profiling, etc. Each component of a given infrastructure has its specialized tool for monitoring, but there are very limited tools which can monitor multiple technologies and devices. One such tool is Prometheus.

Prometheus is an open-source monitoring platform primarily used for monitoring metrics. It collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. It stores data as a time-series database, which can be accessed using a query language called PromQL. Using the metrics collected, Prometheus allows us to observe various parameters both in numerical and graphical format.

Prometheus is often used in combination with Grafana, which is a multi-platform open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics, by using metrics stored as a time-series database in different graphical representations.

Typically, we would like to not only analyze the data in the form of metrics but also make the data collected persistent, which can be done with the help of container orchestration tools like Kubernetes. In this article, we will be looking at how we can integrate Prometheus and Grafana for metrics monitoring while making the metrics collected persistent with the help of Kubernetes.

To set up our work, we need a few pre-requisites:
Cardiac Monitoring Industry Development And Business Growth
Tags: Healthcare, Cardiac, Monitoring, Medical Devices, Business

Cardiac Monitoring Devices Market

Market growth influenced due to following factors-

Increasing prevalence of cardiovascular diseases.

Availability of reimbursement for CM & CRM devices.

Launch of innovative products by market players; increasing investments, funds, and grants for research on CM & CRM devices.

Global market size-

According to the new market research report Cardiac Monitoring Devices Market is projected to reach $26.8 billion by 2025, at a CAGR of 4.0% from 2020 to 2025.

Leading Players-

The major players operating in this Cardiac Monitoring Market are Medtronic (Ireland), Boston Scientific (US), Abbott Laboratories (US), GE Healthcare (US), BIOTRONIK (Germany), and Koninklijke Philips (Netherlands), Asahi Kasei Corporation (Japan), SCHILLER AG (Switzerland), BioTelemetry (US), MicroPort Scientific Corporation (China), Hill-Rom Holdings (US), Nihon Kohden (Japan), BPL Medical Technologies (India), ACS Diagnostics (US), Shenzhen Mindray Bio-Medical Electronics (China), Lepu Medical Technology (China), Shree Pacetronix (India), OSYPKA (Germany), and BTL t(US), Alivecor, Inc. (US), Amiitalia (Italy), Bexen Cardio (Spain), Narang Medical Limited (India), MeTrax GmbH (Germany), and Progetti Srl (Italy).

Top Market Segments-

On the basis of cardiac rhythm management devices by type, the market is segmented into defibrillators and pacemakers. In 2019, the defibrillators segment accounted for the largest market share. This segment is also projected to register the highest CAGR during the forecast period. Factors such as technological advancements in defibrillators, rising prevalence of CVD, and growing focus on providing public-access defibrillators are driving the growth of this market.

On the basis of cardiac monitoring devices by type, the market is segmented into ECG devices, event monitors, implantable loop recorders (ILRs), cardiac output monitoring (COM) devices, mobile cardiac telemetry (MCT) devices, and smart ECG monitors. In 2019, the ECG devices segment accounted for the largest share of the market. The large share of this segment can be attributed to technological advancements such as the incorporation of advanced algorithms and workflow improvement features, such as simplified operations, touchscreen systems, portability, wireless features, and better connectivity with cardiovascular information systems (CVIS), electronic medical records (EMRs), and ECG management systems.

For Details, Download PDF Brochure

Geographic Overview -

On the basis of region, the Cardiac Monitoring Devices Market is segmented into North America, Europe, Asia Pacific, and Rest of the World. In 2019, North America commanded the largest share of the market. The large share of this market segment can be attributed to the increasing incidence of CVD, increasing healthcare expenditure, growing disposable income, growing healthcare awareness, and the availability of technologically advanced devices in the region.
Maximizing VA remote patient monitoring during the COVID-19 response
By: Partners In Digital Health
Tags: Patient Engagement, Remote, Covid 19 Crisis, Monitoring, Medicine

Maximizing VA remote patient monitoring during the COVID-19 response

Objective: The Veterans Health Administration (VHA) has one of the largest remote patient monitoring programs in the United States and is supported by an enterprise-wide infrastructure for providers, clinicians, staff, Veterans, and caregivers. The coronavirus disease 2019 (COVID-19) pandemic, however, presented new challenges: a sudden need to provide large-scale remote monitoring for a new disease that did not yet have a disease management protocol (DMP). Veterans Health Administration (VHA) needed to be ready within weeks to provide this daily monitoring for hundreds — even thousands — of Veterans.

Methods: The US Department of Veterans Affairs Office of Connected Care already had a comprehensive infrastructure in place for its remote patient monitoring — Home Telehealth (RPM — HT) program. Connected Care activated and built on this infrastructure to support providers, clinicians, and staff in their efforts to rapidly bring RPM — HT to Veterans across the nation when they had COVID-19 symptoms or exposure. To do this, Connected Care activated an emergency management plan, rapidly developed a new COVID-19-specific DMP, added weekend monitoring, and procured critically needed monitoring supplies, such as thermometers and pulse oximeters. The strong foundation for Connected Care allowed for innovation and flexibility, such as the training of non-RPM — HT staff in RPM — HT processes, RPM — HT enrollment within acute care settings, and new strategic partnerships.

Outcomes: More than 23,500 Veterans were enrolled for COVID-19-related monitoring from March 2020 to May 2021. During December 2020 and January 2021, the number of Veterans being monitored in a single day topped 2,000. Even with this rapid buildup, patient satisfaction levels remained at about 90% in numerous categories. In addition, the percentage of Veterans admitted to Veterans Administration (VA) facilities while on COVID-19-related home monitoring has been extremely low, at 4%, the monitoring system has been a potential indicator in enabling Veterans who did have the virus to convalesce at home. Further study is needed to determine the impact RPM — HT enrollment for COVID-19 care had on the need for inpatient care.

Conclusion? Read here: https://doi.org/10.30953/tmt.v6.281

Authors
How to Deploy Flink Cluster & Flink-exporter in Kubernetes Cluster.
Tags: Cloud Computing, Flink Exporter, Monitoring, Flink, Kubernetes

How to Deploy Flink Cluster & Flink-exporter in Kubernetes Cluster.

Flink k8s

Preface.

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

There are two Flink’s clusters: Flink session cluster and Flink job cluster. A Flink job cluster is a dedicated cluster which runs a single job. The job is part of the image and, thus, there is no extra job submission needed. A Flink session cluster is executed as a long-running Kubernetes Deployment. Note that you can run multiple Flink jobs on a session cluster. Each job needs to be submitted to the cluster after the cluster has been deployed.

In this article, we will go thru Flink session cluster and Flink-exporter deployments. I will use Minikube.

Part 1. Flink Session Cluster Deployment.

A basic Flink session cluster deployment in Kubernetes has three components:

a Deployment/Job which runs the JobManager

a Deployment for a pool of TaskManagers

a Service exposing the JobManager’s REST and UI ports

I’m using Minikube, so need to execute:

minikube ssh 'sudo ip link set docker0 promisc on'

before deploying a Flink cluster. Otherwise, Flink components are not able to self reference themselves through a Kubernetes service.

Let’s deploy the Flink session cluster. Go to tutorial repository folder and execute:

$ kubectl apply -f flink

The output should be similar to this:

Let’s look at the Flink’s configuration files.

Jobmanager-deployment.yaml

The Deployment definitions use the pre-built image flink:latest, set limits of 512 Mb memory and 0.5 CPU core, defines ports: 6123 for RPC, 6124 for a blob, 6125 for query and 8081 for UI.

jobmanager-deployment.yaml

Taskmanager-deployment.yaml

The Deployment definitions use the pre-built image flink:latest, set limits of 512 Mb memory and 0.5 CPU core, defines ports: 6121 for data, 6122 for RPC, 6125 for a query.

taskmanager-deployment.yaml

Jobmanager-service.yaml

Exposes necessary ports

jobmanager-service.yaml

Ingress-service.yaml

Creates ingress for proxying UI port. Don’t forget to add flink.local to your host’s file.

ingress-service.yaml

Finally, check the Flink deployments status:

$ kubectl get deployment

Output:

Flink deployments status

Navigate to flink.local in your browser: