fabiochiu/medium-articles huggingface dataset:
Your Stress is Not Responsible
By: Sam Morris
Tags: Emotions, Business, Mindfulness, Stress, Coaching

What is your relationship with the word “responsibility?”

If you are like most people, you have mixed feelings when you think of the word. Your mind may go to your job duties, your family obligations, or perhaps your finances.

And as your mind travels to these places, you may have a feeling that lies somewhere on the spectrum between a mild sense of burden or irritation and a full blown panic attack.

We know intuitively and intellectually that responsibility is a sign of maturity. We are meant to take pride in being responsible.

So why do our responsibilities sometimes create so much stress?

To answer this, we need to look at the Latin root of the word “responsible,” which, when taken literally, means, “able to respond.”

As we take on certain responsibilities and obligations in life, we frequently do not yet have the ability to actually respond to what we are taking on effectively. So there is a cognitive dissonance in our minds between the choices that we have made and what we are currently capable of doing. Throw in a slew of unknown circumstances and you have the perfect recipe for stress.

And then what do people tend to do to cope with the stress? They frequently add pride to the concoction, as though the stress they have taken on is somehow a badge of honor.

Pride may tell you that you are stressed out because you have so many responsibilities, when, in fact, you are stressed out because of all the obligations that you are unable to respond effectively to.

The unconscious mind will learn to associate responsibility with both stress and pride, and you will likely end up seeking and attracting more stress as a result.

Rather than truly being responsible, you will be emotionally reactive, because stress needs an outlet. Then you will blame the reactivity on your responsibilities (which, of course, were the things you were actually not able to respond effectively to).

The “pride-stress-responsibility” association has become so ingrained in our culture that many people believe that it is an inescapable part of being a high-performing human, when, in fact, it is not only escapable, it is our responsibility to escape it!

After all, excess stress can only lead to poor job performance, poor health and an unfulfilling life. Truly being responsible means doing a daily assessment of our present capacity from an honest perspective.

And when we do take on more than we can presently handle, which we often need to, we must have the humility to accept that we simply have not yet developed the ability to respond to a particular situation with a sense of grace.
Why meditate? How it can drastically improve your life?
By: Aahlad Kethineedi
Tags: Lifehacks, Stress, Meditation

Meditation is very essential in one’s life. With exponentially growing world, obviously there exponentially growing stress rate to deal with.

This read won’t take up much of your time. I know you are well aware of why you should meditate and how it can improve your life. Sorry for the clickbait-y title :).

The only two methods of meditation that are effective to me as I have a fast paced life are Transcendental Meditation and the Do nothing meditation.

Keep in mind this is just to relax and not for spiritual awareness and become a guru. You might have to read someone else’s Medium posts for that.
So, I always wonder just how many people out there relate to me… It’s a big world and many of us…
Tags: Teens, Study, School, Stress, Overthinker

So, I always wonder just how many people out there relate to me… It’s a big world and many of us surely do have things in common, similar views, as well as outlook on life. I’m a student. It’s probably the prime years of our lives to go out and have fun, bearing little weight on our shoulders, or is there more to that? In truth, I’m an over-thinker. There’s just so many things I keep thinking about, dwell on, only to revisit old thoughts yet again. It’s not that severe to the point that I struggle a lot with daily tasks but it does take a toll on my emotional wellbeing, stress levels and happiness meter. It may be because I just care too much, I wish I could care less. I want to do my best, but I can never be sure if it’s my best. I want to believe in myself, but can I really do it? There are so many things to care about, school, family, friendships, work, other commitments. I am also an over-achiever or to say, I set really high expectations of myself. Not that they are ridiculously unachievable, but it’s really hard, all the time. I just can’t seem to take it easy on myself almost all the time? It’s healthy to a certain extent but too much of it can really add on unnecessary stress into the already stressful teenage years. However, there are also times when I’m in a rut and just can’t seem to stop procrastinating. It’s such a common problem we all share but yes, I fall into that pit at times. I just wonder if there are people that are like me! I’m currently waiting for my results and I’m aiming for straight 9 A1s. I studied really hard and hope I get there. It’s a real personal goal and yes, this is the perfect example to illustrate my thoughts above. First writing and I’m looking to writing more widely and improve on my writing! Do share with me any tips on how to improve!
What is the Scalability in the distributed system?
By: Yallappa Deveendra
Tags: Distributed Systems, Blogging, Writing, Design, Scalability

Scalability of system.

Scalability (image source — google)

Scalability of the system means to handle the task as the system grows in size, by growth can be increased in the users or increased amount of work. Like a number of transactions increased, the system should give the correct response irrespective to the increased user transactions or network growth.

Different ways to scale the distributed system

Horizontal and vertical scaling: Horizontal scaling is achieved by adding more server to the existing servers, where Vertical scaling is adding more more resources like (RAM, CPU, Storage etc.) to the existing server.

Vertical vs Horizontal (image source — google)

With Horizontal scaling much easier by adding more servers to the existing resource pool, where vertical scaling is updating smaller servers with bigger capacity servers which involves the downtime for the system.

Good example for vertical scaling are Cassandra and MongoDB by which we can add more machine to scale. Similarly, good example for vertical scaling is Mysql. Is easy to scale by changing smaller machine with bigger one but it involves the downtime.
System design paradigm: Sharding
Tags: Distributed Systems, Sharding, System Design Interview

Scale out for too big data or too many writes

Primary-replica pattern and in-memory cache solves the problem on the read side. But what if the total data is too big to fit in one host or the write throughput is too high? The answer is very intuitive: we split the total data into multiple hosts. This is called sharding.

Notice that write latency is usually not an issue because

We expect write to be slow for most applications

The write can return immediately after persistent on commit log, we therefore trade strong consistency for lower latency

Sharding key

A key decision of sharding is given a record R, how to determine which shard it belongs to. The sharding design contains a key function that maps R to a key string. We then compute a hash of the key. The modulo of the hash(integer) on N is the shard ID. N is the number of hosts.

Shard Hash(Key(R)) mod N

Notice that Key(R) doesn’t have to be available at query time. For example, R is <user_id, order_id, quantity>. We can use order_id as a sharding key(Key(R) = order_id). We can still serve the query of finding all orders for a given user by doing the query on all shards.

When choosing sharding key should consider

Avoidance of write hotspot

Avoidance of query hotspot

Query pattern: one or all

There are two query patterns.

Query One: If we are querying records with a single sharding key, the query can be handled by the single shard.

Aggregation: If the query’s predicate can’t be translated to a subset of sharding keys, then the query must be processed on each shard. The result is aggregated before returning to clients.

Each pattern has advantages and weaknesses.

The weakness from each pattern can be mitigated. For example, reading hotspot can be mitigated by replication and caching. Read latency can be mitigated by imposing shard timeout on RPC(in expense of data completeness).

Query hotspot is interesting in that aggregation doesn’t really distribute the query load, but simply repeats the load on each shard. This way, the query loads on each shard are the same. This makes it possible to assign the same number of read replicas per shard.

The choice should be determined per application tradeoff.

Last words: don’t do it unless you must

Sharding is rarely a good idea. The costs include

Harder to write application code

Harder to reason and debug the system

Harder to operate/maintain

Higher order of complexity on everywhere

Tremendous architectural work is needed to get sharding right and convenient for application developers. So don’t use it lightly. The good news is that most projects don’t have to be shared because the write QPS is very low in the first place. When teams do need sharding, they should be mature enough to know what they are doing.

Future reading

Airbnb’s sharding story
Fault Tolerant Virtual Machines. A summary of “The Design of a Practical…
By: Dan Allford
Tags: Distributed Systems, Virtualization, Fault Tolerance

Disaster recovery is a key part of application architecture. In the case of failure of a machine, we would like something better than complete loss of the application. A common modern application design principle is to use stateless applications sitting over reliable/replicated data storage (eg/ AWS S3 or HDFS). However there are numerous reasons this may not be possible (eg/ legacy applications). Given that legacy applications run the world, we would like to be able to improve the failure scenarios of these stateful applications. Often this will be done through regular snapshotting of VMs, although these snapshots will always be at least some what out of date, and will require some time to complete the restore process.

By using a “primary-secondary” (active-passive) set-up for virtual machines the authors enable failover from the primary to the secondary without service interuption or application modification.

With this system, a user of the virtual machine should not be able to detect (aside from an increased latency) that the hardware on which the VM is running has crashed.

The resulting system is benchmarked to incur less than a 10% performance penalty compared to a single, non fault tolerant VM.

The paper only deals with single processor systems, and although the ideas could be extended to multi processor systems, a naive extension would incur massive performance penalties (due to the need to serialise concurrent memory access).
Redis Cluster – Benefits of Sharding and How It Works
By: Tugberk Ugurlu
Tags: Distributed Systems, Database, Sharding, Redis

Redis is by far one of the most frequently used data stores. It’s fascinating how much our our software-developer-minds go to Redis when we are faced with a data storage problem that requires some level of scale. Even if this might make us feel guilty, I have a somewhat confident assumption that this’s the case, and there is probably a relation here to its simplicity: e.g. Redis is ‘just’ a data structure server, a hash table in the ‘cloud’, etc. (I know I am a bit exaggerating here, but hopefully you get the idea). Redis also makes digestible and reasonable trade-offs, and it allows us to solve many problems which require certain degree of scale.

For a long time, Redis has come with an out-of-the-box replication functionality, which allows for a high availability (HA) setup as well as allowing us to scale the reads by distributing the load across replicas with the cost of eventual consistency. However, it was only in April, 2015 that Redis added support for a built-in sharding functionality with its version 3 release. I have been working with several Redis Cluster setups for a while, and have probably read the Redis Cluster spec at least couple of times. In this post, my aim is to give you more understanding on what problem Redis Cluster actually solves, why you need such a setup, and most important details you need to know about its configuration and implementation details based on my own experience.
Bounded context microservices: Alchemy of the current age
By: Najmus Saqib
Tags: Distributed Systems, Microservices, Architecture, Observability

You must have heard about alchemy, the predecessor of the field of chemistry. Alchemists were trying to create gold from ordinary metals, philosopher’s stone, or the elixir of immortality. For centuries alchemists made efforts to achieve these tremendous goals. They were not able to achieve these goals but the discoveries they made and the practices they invented paved the way for modern-day chemistry.

In the software development world, something similar happened. The ultimate promise of microservices architecture is bounded context services where the developer of one service doesn’t have to care about the business logic and data of other services. This is the utopian state that any organization aims to achieve when it starts breaking its monolith into microservices. In reality, boundaries are not very clear in different domains of a monolith. This results in services doing lots of data sharing and distributed transactions which essentially turn them into a distributed monolith.

The best advice for architects who want to do transactions across services is: don’t! — Fundamentals of Software Architecture: An Engineering Approach by Mark Richards and Neal Ford

Similar to alchemy, microservices architecture was not able to attain its ultimate goal i.e. bounded context services but it paved way for a much bigger phenomenon. It helped developers to get ready for the era of cloud computing and Software as a Service. Working on distributed systems requires a much more sophisticated expertise and toolset. As they say

In distributed systems every bug is a murder mystery.

Thanks to microservices now we have observability tools and patterns that help us trace bugs across distributed services. Devs are now much better equipped to handle APIs and SaaS tools. The microservices architecture was the catalyst in the adoption of containers and serverless.

So next time when you hear someone bashing microservice architecture for adding complexity in web apps without fulfilling its promised benefits, do tell them the fruits that it has given us.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬
my smart environment to get started with deep network and SAR images
By: Ghomsi Adrien Arnaud
Tags: Sentinel Hub, Distributed Systems, Radar, Machine Learning Ai, Geospatial Intelligence

1) A smart operating system

Among the gnu / linux distribution that exists, the one we will use to build our environment is called UMI. UMI pronounced “You Am I” is nowadays an operating system that offers technology through a philosophy of “know-how” and “know-how”, identifying with the needs of the user. Ideologically, UMI stands for “I am you” and stands out as the solution that exists because of who we are.

Install UMI

Download UMI here

Create a bootable USB stick

Install UMI OS

2) A smart environment installation

Install Anaconda

Designed for data science and machine learning workflows, Anaconda is an open-source package manager, environment manager, and distribution of the Python and R programming languages. the digital ocean community has a good guide on how to install it on a linux OS. below some regular basic commands that you will use, at least I strongly recommend that you always refer to the conda documentation in case of difficulty:

Basic commands

Using with shell: Don’t forget to first use this command in your terminal (moon-terminal in umi-os) after installing conda in order to initialize it.

$ conda init

Create Anaconda environments: For example, a Python 3 environment named smartSAR_env

$ conda create --name smartSAR_env python=3

Activate Anaconda environments: For example, environment named smartSAR_env

$ conda activate smartSAR_env

Deactivate Anaconda environments: For example, environment named smartSAR_env

$ conda deactivate

install package on my smartSAR_env: For example, a sentinelsat API.

$ conda install -n smartSAR_env sentinelsat

Install Docker

Docker is a container or is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.

3) A smart tools installations

Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.
Let’s Bulky WordPress Project Run On a Cloud With Galera, Nginx, Redis, HaProxy, and Varnish
By: Davood Falahati
Tags: Haproxy, Distributed Systems, WordPress

Photo by Launchpresso on Unsplash

WordPress is an old but handy CMS available still in 2020. I am not a fan of WordPress, and to be honest, I never use it for my personal or professional purposes. However, there are many cases that I see the WordPress is still the best solution for medium and small businesses that are willing to launch their own website in the shortest time. Even in 2020, I still suggest the very WordPress to everyone who doesn’t want to suffer finding developers to create and maintain their websites and spend tiny sums for their “hello world” businesses online.

Amid one of my advisory jobs, I came across an unrelenting situation in which our WordPress-powered website was devouring server resources. We were barely able to serve 100 concurrent users. That was a big failure for my company’s marketing strategies. You are right, we could migrate from WordPress to a much faster technology stack such as Golang and Reactjs, but it was not palpable for the company I was working with. Instead, we went for building up our resources. We had some fair virtual machines and we decided to use them up in a cluster structure. We used 7 machines and I want to share my experience with you in solving our problem.

We use 8 servers with the following details:

front.example.com hosts load-balancer (and you can say WAF)

wp-1.example.com hosts 1st WordPress codes and its underlying contents equipped with php-fpm engine. This machine has a web and a cache server installed locally.

wp-2.example.com bears a replication of WordPress and php-fpm. This one has its own nginx and varnish servers as well.

cache.example.com this one hosts redis server and let wp-x servers to read/write their sessions on it.

file.example.com: This is where we put our WordPress wp-content folder on. We let wp-x servers to have an access to its content through NFS.

db-1, db-2,db-3.example.com: these three servers let would create your Galera cluster and wp-x servers would be able to connect to them.

Ubuntu 18.04 is installed on all 8 hosts. By the way, you can use any OS you want.
Hash Tables In Distributed System
Tags: Distributed Systems, Data Structures, Distributed Hash Table, Hash Table

In our case we need a consistent hash function. Let us define our hash function h(x) = x mod R , where R is a number such that R > n

Where n is the total number of nodes.

We see that range of x mod R => 0….R-1

Each node now stores hash values between a range

When R = 20 and n = 4

When we need to find the value against a key K, we calculate h(k) and ask the node corresponding h(k) for the value.

Currently we are not concerned about replication or node addition. A trivial redundancy could be created by having each node also be responsible for secondary range. In case of replication at 3 nodes, each node now has primary, secondary and tertiary ranges. This could be extended till n replication.

For addition of a node, the node joins the group. The range is split from one of the nodes, and the new range is provided to the new node. The node is currently shadowing and will copy all keys data from primary node for the range. Once the copy is complete, the split is broadcasted and the new node processes for new range.

Adding a node

Addition complete

Removing a node is similar, the secondary node takes responsibility of the range when a node is removed / is no longer responding. It becomes the primary owner for the range of hash values.
Trace api request logs in Rails
By: Kavita Jadhav
Tags: Logging, Rails, Distributed Systems, Splunk, Debugging

We all know benefits of logging. It enable developers to quickly troubleshoot issues even before reproducing it. These can be logs generated by framework, custom logs added by developers.

In Production environment we have many app servers running behind the load balancer. Request can be processed by any server and hence to get the request logs you need to know which server has processed the request. It gets even more complex when you have multiple services in place.

In distributed systems, computing logs are stored in shared drive or sometimes on distributed servers itself. Later these logs are aggregated in tools like Splunk, LogDNA, Logstash to make them accessible at single place and enable users to search/read logs.

Since now all the logs are aggregated at one place it reduces the effort of checking logs on multiple servers but still you have to search for keywords(like api_request, query parameter, custom log statements) to get expected logs. Using this approach you might loose on some important logs which might increase your debug time.

Example: Here I have created an application and added /health endpoint in it. To check logs am searching for text /health. This is giving me all log statements having keyword as health in it. Also if you look at the timeline section you are getting older logs as well. Hence you have to manually compare time as well to get specific request.

log search results with keyword

This can be solved in rails by using rails_distributed_tracing gem written by Ajit Singh.

To get started add rails_distributed_tracing gem to your Gemfile and run bundle install command. Then follow the steps mentioned in README file.

gem 'rails_distributed_tracing'

Now restart the application and send a request from browser or postman and monitor network call logs. You will get X-Request-Id attribute in Response Headers as below:

Api request network logs

Now you can use X-Request-Id to search all logs generated for api request in all services.

log search results with request_id

Here you are getting all logs for specific request only. Please note that I am using a different request_id to search logs as I have generated these at different point of time. You can search logs with same request_id available in response headers.

I hope this will help to make debugging a easy task!

Happy coding!!!😇
Some Thoughts on Microservices Architecture
By: Andrew Glynn
Tags: Systems Thinking, Distributed Systems, Design Thinking, Microservices, Architecture

Microservices architecture is not a bad thing, microservices offer inherent horizontal scalability, decomposition into small parts, etc., but microservices architectures are also the fastest way to lose your shirt if you don’t think fully through what you’re attempting to accomplish.

Put it this way: what are the major differences between a complex distributed application and a large-scale orchestration of microservices?

· The base protocol of most microservices, REST, is one of the most limited protocols available, particularly in terms of recoverability from network errors.

· There is no developed, proven framework for orchestrating microservices, whereas the Actor framework in distributed applications is well-known, well-proven and available today on a number of platforms beyond ERLANG and Smalltalk.

· The pieces of a distributed application were designed with that application in mind.

· Some thought has usually been given as to how the overall state of the application is to be maintained, while many microservice developers and management proponents seem to think that it will magically maintain itself.

· And the biggest difference? Terminology, nothing more.

Since it’s fairly well known that complex distributed applications are a difficult undertaking, the fact that many people in the software industry seem to believe a large-scale orchestration of microservices to produce a complex resulting application is simple is ridiculous.

The most commonly touted MSA I’ve seen gives no thought to shared storage or eventing, which should be the first two things settled on before attempting any distributed architecture. MSA in many ways does simply take SOA and go further, but that further doesn’t make it easier, it makes it more difficult. Done properly, it should also offer better horizontal scalability, more flexibility when refactoring, etc., but those don’t come for free.

The easiest way to begin thinking about it is that with your finished microservices you have a set of reusable functionalities, but you don’t yet have a system. How you go about building a system from that set will determine in the end whether you have a flexible, maintainable architecture or a stress-inducing money pit that accomplishes little to nothing.

The quickest answer is to look at the required functionality for a complex distributed application, because in essence, that’s what you’re creating. The simplicity of the services themselves only moves the complexity up the stack. From that functionality you can determine the state that needs to be maintained by each service and the shared context that all the services will need access to. Finally you need to make a decision on what will have the capability to modify that context and how you’re going to make it easily and quickly available to every service involved in the orchestration.
Building a Deduper, a systematic approach in our system context
By: Trinath Patri
Tags: Engineering, Distributed Systems

If you are a software engineer, you would have heard about deduplication hundreds of times. Every system that wants to scale or focus on scaling should be interested in filtering out duplicate processing as early as and in the shortest possible time in the event processing pipeline. At Walmart Global Tech, we are no different for our enterprise rule engine platform that was built to optimize a specific event (we call it as override on rules) where we needed processing that was created and updated in many different ways by the user (e.g., UI/API/spread-sheets via REST or Kafka).

Assumptions:

If a duplicate is found, the processing will be considered as a success.

If the duplicate check fails, we will continue with processing.

Approach #1: (Sprouting)

My first system design approach was to rely on two main components such as:

Bloom Filter (that determines either MAY BE present or NOT present) A key value store (MemCached or Cassandra(C*) (as shown in figure 1 below)

We would first check if an override’s key (hash of it) may be present in the bloom filter, then call the second key/value store (cache) to find a duplicate. If a duplicate is not found in cache or a key was not present in the bloom filter, then processing would follow the normal flow. If a duplicate is found, the system will mark the event as “duplicate” and respond to the caller marking it successful processing. Here is a high-level system diagram of our override processing system (shown in figure 1).

Figure 1 : System Diagram for ‘Override Deduper’

Advantage:

It’s an easy design and simple to understand. As the data volume is not so large yet (although it’s growing), we did not focus much on a cache invalidation strategy. We preferred to use Write Through Cache (latency is around ~1ms for the same data center(DC) and <10ms for cross DC for cache write and there is no additional cost to write to DB as earlier we were writing to C* anyway) over Write-Around or Write-Back Cache.

Challenges:
NATScaled : Distributed Services on TailScale Network
By: Naveen Anto
Tags: Distributed Systems, Nats Streaming, Golang, Tailscale

Toy app running multiple NATS servers in a TailScale Network

The End Product

The aim of this toy app is to experiment with Tailscale (more on tailscale below). When we run the application (a binary actually), it will :

Ask for person to login via Gmail/Microsoft/Tailscale Once logged in, will start a server Repeating Step 1 & 2 in multiple systems (located anywhere) with same login account will result in all those servers communicating securely with each other i.e, these servers will form a cluster.

Diagrammatically, after running it in three machines, it’d look something like this:

Fig 1: Three NAT servers (machines) are connected to each other (peer-to-peer) using Tailscale. [Arrows are bi-directional but are drawn unidirectional for aesthetics]

Tech Stack: Go, TailScale, NATS

Now lets try peeling this layer by layer

NATS

Imagine a situation where you as a developer is tasked with creating a service that publishes events (user events) that are consumed by multiple services (analytics, notification). If you are planning to use NATS package this is what you’ll go through

Start a NATS server in a machine Create a service that uses NATS publisher client to publish messages to a topic Create subscription services using NATS subscriber to subscribe to the published topic

Fig 2: A NATS Publisher client publishes under the topic “foo” and 2 NATS subscriber clients are listening to events under “foo”

In a production environment, to not have a single point of failure and for many other reasons, we’d have more than 2 servers. NATS server supports combining multiple machines to form cluster, allowing servers to communicate with each other (Fig 3).

Fig 3: Same as Fig 2 but with multiple NATS servers forming a cluster

In-order for NATS servers to form a cluster, they need to know the IP address of at-least one other server in the existing cluster i.e, every time we (re)start a server we need to seed it with IP information of any other running server in the cluster. This process of identifying other services is usually referred to as service discovery.

The task becomes more cumbersome if the servers are assigned dynamic IP addresses each time. There are various solutions available to solve this but we are going to experiment with Tailscale (highly unconventional!).

Tailscale

Tailscale creates a private network between devices (or machines) i.e,

You could install Tailscale app in your laptop, mobile, cloud server, raspberry pi, brain’s cerebellum [not serious here!] etc… Login with Google/Microsoft/Tailscale All of these devices will be connected via private network.

Fig 4: Borrowed image from tailscale showing how you could connect directly to multiple machines without VPN gateways or jumpboxes

People who are interested in its working, please read this awesome post.

In addition to providing a secure peer-to-peer private network, tailscale also makes sure that each device is assigned a unique Tailscale IP address that remains static for the account signed in with.
Handling user authentication and authorization after load balancing your web app.
By: Dan Mitreanu
Tags: System Administration, Distributed Systems, Web Design, Web Development, Load Balancing

Load balancing your web application is the way to go when expecting or experiencing a large amount of traffic, but when it comes to designing an authentication and authorization flow it can become more complicated on the back-end side.

Why load balance?

In high traffic scenarios, such as hosting a social networking website, a popular blog, an API or just about any app on the Internet with a large number of valid incoming requests it can be overwhelming for the physical server to handle all of these requests and produce the desired responses.

Continuing I will be discussing the apparent and deeper implications of scaling up your authentication and authorization flow (auth from now on) handled by your web server into a system which cannot guarantee that subsequent requests from the same client will be handled by the same server.

Before we start…

If you are reading this you probably have got an idea what load balancing is, but for the sake of this article, I do feel the need of making sure we are all on the same page.

What is load balancing?

In short, load balancing is the process of distributing traffic across multiple, different server instances using a particular load balancing algorithm. Think of it as a cop standing in front of a toll plaza that redirects you to the toll booth with the smallest queue.

Toll plaza, courtesy of tribuneindia.com (https://cmsimages.tribuneindia.com/gallary_content/2020/4/2020_4$largeimg_1893155391.jpg)

Let’s assume that your original server provided by your hosting platform can handle a maximum of 1,000 requests per second before the experience of waiting for a response becomes frustrating. You are flourishing and users are starting to use your app more and more, hence the traffic. You wouldn’t want them to grow frustrated, would you?

So what do you do? You implement a load balancer. You either pay for an instance provided by AWS or whichever service you use, or you take the time to configure your own. But a load balancer won’t work on its own as if it’s magic. You need multiple servers so the balancer knows where to redirect your request so it is processed, and this becomes your new problem when it comes to auth flow. Why? Take a look:

Load balancing an API

As you can see, even though Bob sends two requests to the same API in a very small time interval, his response is still being handled by different server instances. This example goes to show that you absolutely cannot predict which instance will ultimately handle your client’s request. And even if you can, I doubt you can reliably do so with a load balancer that does its job properly.
Snowflake Deep Dive
By: Bigdata, Cloud Practice
Tags: Data Warehouse, Distributed Systems, Snowflake

Abzooba is an AI and Data Company. BD&C Practice is one of the fastest growing groups in Abzooba helping several fortune 500 clients in there cognitive journey
Microservices — Design and Development
Tags: Distributed Systems, Api Gateway, Microservices, Microservice Architecture

Microservices development stages

1. API gateway

2. Inter-Process Communication

3. Service Discovery

4. Event-Driven Data Management in Microservices

5. Deployment of Microservices

API Gateway:-

In a monolithic application — one set of endpoints are replicated, with load balancing used to distribute traffic among them. In microservices — Each microservice exposes a set of fine-grained endpoints. To handle multiple endpoints we use an approach called API Gateway.

API Gateway — a server that is a single entry point into the system. Encapsulates the internal system architecture and provides an API that is tailored to each client.

Other responsibilities:-

a. authentication

b. monitoring

c. load-balancing

d. caching

e. request shaping and management (request routing, composition and protocol translation)

f. static response handling

All requests from clients first go through the API gateway. It then routes the requests to the appropriate microservice. API gateway handles the request by invoking multiple requests and aggregating the results.

Examples of API gateway in JVM — Netty, Vertx, Spring Reactor, JBoss Undertow

Non-JVM environments — Node.js

NGINX Plus can be used as an API gateway. Scalable, high-performance web server and reverse proxy that is easily deployed, configured, and programmed. It can also manage authentication, access control, load balancing requests, caching responses, and provide application-aware health checks and monitoring.

Reactive Programming model:- To write the API composition code using the async callback approach is messy. It's better to write API gateway code in a declarative style using a reactive approach.

Examples — Future in Scala, CompletableFuture in Java8, Promise in Javascript.

Reactive extensions (Rx or ReactiveX), RxJava for JVM, RsJS for JavaScript



Service Invocation:-

1. Async messaging based mechanism(message brokers like JMS or AMPQ)

2. Sync mechanism (HTTP or Thrift)

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

2. Inter-Process Communication:-

1. One-to-one

2. One-to-many

Second dimension

1. Synchronous

2. Asynchronous

One-to-one and Synchronous — Request/Response

One to one and Asynchronous — Notification (or) Request/async response

One-to-many and asynchronous — Publish/Subscribe (or) Publish/async responses
MDC Tenant Copy within a cluster - SAP HANA
Tags: Sap Hana Mdc, Distributed Systems, Database Administration, Sap Hana, Tenant Copy

Below, we can see all the processes running for ABC host is actually distributed across nodes — host1 and host2 ( simply, this database was setup with the name ABC, and as you know this is the default and first tenant in this container, since we added the host2 as worker node while setup this is distributed):

Now, we can navigate to $SECUDIR to generate a PSE key:

Using inbuilt sapgenpse commandline util, we can generate the key (with any name, here we name it as source.pse):

Download py script — extract_certificates.py from SAP Note 2175664, and pass new PSE file as argument and execute it. You’ll see a big chunk of key encrypted values (for readability I’ve removed few lines):

Connect to SYSTEMDB and create a PSE store

Make sure to turn on multiline [\mu] from hdbsql console

Now, apply the generated certificate to the newly created PSE store

Set the purpose of PSE to SSL since we communicate between the hosts in a secured layer
Le Pont Avalanche-Ethereum : Ce Que Vous Devez Savoir
By: Avalanche France
Tags: France, Distributed Systems, Interoperability, Cryptocurrency, Blockchain

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Consistency Model in Distributed System
Tags: Distributed Systems

in The Next Team
Layered protocols in distributed system
By: Digital Classes
Tags: Distributed Systems, Protocol, Computer Architecture, Computer Science, Layered Protocol

Due to the absence of shared memory, all communication in distributed systems is based on message passing. When process A wants to communicate with process B, it first builds a message in its own address space. Then it executes a system call that causes the operating system to fetch the message and send it over the network to B. Although this basic idea sounds simple enough, in order to prevent chaos, A and B have to agree on the meaning of the bits being sent.

Many different agreements are needed. How many volts should be used to signal a 0-bit, and how many volts for a 1-bit? How does the receiver know which is the last bit of the message? How can it detect if a message has been damaged or lost, and what should it do if it finds out? How long are numbers, strings, and other data items, and how are they represented? In short, agreements are needed at a variety of levels, varying from the low-level details of bit transmission to the high-level details of how information is to be expressed. To make it easier to deal with the numerous levels and issues involved in communication, the International Standards Organization (ISO) has developed a reference model that clearly identifies the various levels involved, gives them standard names, and points out which level should do which job. This model is called the Open Systems Interconnection Reference Model, usually abbreviated as ISO OSI or sometimes just the OSI model.

The physical layer is concerned with transmitting the Os and 1 s. How many volts to use for 0 and 1, how many bits per second can be sent, and whether transmission can take place in both directions simultaneously are key issues in the physical layer.

Read Complete article click here
My quick dive into Unison
By: Duane Bester
Tags: Unison, Distributed Systems, Functional, Concurrency, Programming

Run all Counters

At this point we can do something silly and wire up a main function that deploys our counters:

⚠️ Do not run the above ⚠️

If you were to run the main function by doing run main . You’d have trouble stopping your threads, even after "Done" from main was printed. 😬 You’d have to stop the ucm process.

How can we have better control to clean up resources? On interrupt or program completion we want something to stop all those threads we just launched.

Controlling Threads

Bracket the Counters

bracket to the rescue! Also lifted from Haskell this function gives us a nice way to acquire, release, and perform some body of work.

The type signature of bracket is a little confusing, and I recommend reading through the ability docs to become slightly less confused:

.> view bracket io.bracket: '{IO} a -> (a ->{IO} b) -> (a ->{IO} c) ->{IO} c

io.bracket acquire release what = ...

Notice bracket takes three functions; acquire , release , and what . And the output of acquire needs to match the input of release and what .

It’s safe to assume we will move our forkCounters into the acquire function and therefore acquire will return a list of ThreadId . Which means our what and release functions need to accept a list of ThreadId . This makes sense as we definitely want release to be able to stop all those threads!

What
DynamoDB and distributed lock
Tags: Dynamodb, Distributed Systems

Text version of this post can be found here.
Blockchain dan Microservices
By: Rafi Raihan
Tags: Distributed Systems, Decentralization, Blockchain, Emurgo, Microservices

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
用 Caffeine 和 Redis 管理 Cache 案例分析
Tags: Distributed Systems, Cache, Redis

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Tips On How To Be A LinkedIn Celebrity Even If You Are A Nobody
By: Dennis De Silva
Tags: Illumination, Entrepreneurship, Business, Social Media, Marketing

Tips On How To Be A LinkedIn Celebrity Even If You Are A Nobody

Photo by Greg Bulla on Unsplash

I have been a lurker on LinkedIn since I joined back on July 15, 2009.

Back then, I would use the platform as an online resume. And update it now and then — to impress recruiters with the hope that they would call me for an opportunity.

During my carefree days, I would temporarily stop using LinkedIn when I just got a new job then re-establish its use once again when I get bored with my work at some point in time.

I use the platform to search for jobs, connect with recruiters that would possibly hire me. I also use it to read articles about how to grow my career for the past nine years.

I also began to stalk profiles of thought-leaders known in their respective industries and read whatever they have in their feed.

Upon browsing my newsfeed- I also notice profiles that seemed like an ordinary guy like me but had upheld a voice in their respective communities.

It seemed that they are just themselves by sharing their message.

I told myself that I want to be like them — speaking their truth by documenting and publicizing their experiences every day.

I wanted to share my life lessons and make a difference in my little way.

But fearful thoughts started to creep.

I started asking myself several questions.

How can I encourage people to listen — I am not famous or a known authority in my chosen field?

I am not a corporate C-suite executive — what if people criticize the things I would be saying?

Maybe I would probably withhold putting my message out there since I might look like a trying hard scumbag who is just clamoring for public attention.

After all these thoughts running into my head, I still went ahead to share my content.

I will go out there and put my message out there. — Never mind the likes or the amount of engagement that I would be getting ahead.

I made a decision two years ago to be bold and consistent in putting my message out there.

I was surprised by the attention that I am getting right now by just sharing my experiences.

By posting online as a steady routine, I was even awarded — by an Asian online magazine as one of the Top 100 Filipinos to Follow On LinkedIn for Inspiration and Learning back in December 2019 and this year — 2020.

It was a life-changing event — I am overwhelmed with the engagement and opportunities I now receive from LinkedIn.

Whether you want to thrive your business or reinvent your career, I hope this article can help you lead a fruitful journey — on LinkedIn.

I would be sharing the tips I learned during the past few months of being active — and the best part is this guide will not cost you anything.
Volume Eight of The Best Illumination Writers Christmas Reads
By: Liam Ireland
Tags: Christmas Gifts, Illumination, Short Stories And Poems, Writing

Volume Eight of The Best Illumination Writers Christmas Reads

Photograph by Ian Schneider on Unsplash

Just when I thought it was all over, the Christmas themed submissions that is, still I see some really nicely written and highly relevant articles flowing into the queue. So, for as long as I can see them coming in I will continue this series, now on volume eight, no less. Thank you every single one of you for such illuminating pieces, which other readers tell me are very much appreciated during these difficult days. Keep it up guys an gals, I love it.

Elizabeth John

Kenan Kolday

Cinto

Vivi Ambelioti

Abrakadabra

Jay Krasnow

Adelia Ritchie

Jim McAulay🍁

cj lone

Floyd Mori