fabiochiu/medium-articles huggingface dataset:
Life is like Jazz () (weird :)
Tags: Oceans, Baby, Sharks, Life, Jazz

Hmm.. this place seems pretty simple from blogger yeah the interface is quite impressive. I saw few of my friends using this platform so i thought of giving it a try and also this morning i’m in one of mine philosophy mood so yeas i thought what better could be to write a medium story i guess talking is allowed here cause that’s how i write content i talk and tell stories (pretty dramatic ^^).

So, i think by now if you read it till here i have your attention. So what do you think about the tittle i think it got to be a lot better cause we are talking about life here but don’t you think there’s quite a bit of similarities b/w life and jazz well i can’t understand both of ’em :/ well if you’re bored from your daily routine from all the things that are happening around you why don’t you change the octave you can come back to the base scale later try walking a melody on the higher notes or better yet go down you don’t always have to be at the same place switch b/w them. Play a love song or Run a marathon do something if you feel like going away go away cause yea! life is like jazz . (Always trying to Improve ;)

I personally feel i could have written better but still managed to scribble something (Really need emojis here!!) I hope if you took some time out to read it you’ll leave a reply or comment of some sort :E . I’ll come here anyways just need to be in that zone and yeah i think i like the way i write but just need to write more stuff have to put some dedication to write my brain out here cause i usually have many things to write about. Haha! read it till the end

HAVE A GOOD Day!
#67: The Ocean Cleanup
Tags: Oceans, Joe Rogan, Daily Blog, Boyan Slat, Plastic

You can tell he’s going to be a huge thing in a couple of years by the way he speaks; and by the way he referred to the Great Pacific Garbage Patch as a “Good Starting Problem.” — Comment on the podcast Joe Rogan Experience #1402 — Boyan Slat

When you search for “Plastic in Oceans” via Google Images you will probably see something similar like in the image above. Also when you travel to places near the ocean or live near the ocean you have probably noticed something similar, maybe not that crazy like in the images but if you went diving, surfing or just swimming you probably encountered plastic in the water.

Before I watched the podcast episode of Joe Rogan with Boyan Slat I thought that plastic and the ocean is a problem where you can’t find a sustainable solution to fix it, but the project The Ocean Cleanup of Boyan Slat gives me an optimistic feeling that a solution will be found in the near future. Quoted from Wikipedia:

The organization conducts scientific research into oceanic plastic pollution. It was founded in 2013 by Boyan Slat, a Dutch-born inventor-entrepreneur of Croatian origin who serves as its CEO. It has conducted two expeditions to the North Pacific Gyre, and published scientific papers. The clean-up approach uses barriers in ocean gyres to scoop up marine debris as the barrier is pushed by wind and current. The project aims to launch a total of 60 such systems in the patch by 2021. They predict this capability could clean up 50% of the debris in the Great Pacific Garbage Patch in five years. In late October 2019, TOC announced a new initiative, The Interceptor, to attack the trash problem closer to the source, with plans to prevent the 80% of ocean trash coming from 1000 rivers worldwide.

With this post I just want to spread the word for this amazing project! If you are interested in more detail just watch the podcast episode:
The Scalability Issue Remains for Bitcoin and Ethereum
By: Emogi Coin
Tags: Scalability, Ethereum, Bitcoin, Cryptocurrency, Blockchain

The latest round of FUD that really seems to be making its way around the news cycle seems to be focused on the scalability issue. Whether its talk about the Ethereum network being at capacity thanks to all the new ERC-20 Tether (USDT) transactions, or the continued struggles of second-layer scaling solutions like the Lightning Network, the ability for the top cryptos to take on extra traffic is currently limited.

So what does that mean for the crypto space as a whole as we move into a period of time of increased global financial uncertainty?

The fact that this was also a concern during the bull market of 2017/2018, with not much changing in the way of solutions to these issues for Bitcoin and Ethereum, shows that these projects may not ever be able to solve these problems while keeping true to their original design.

That opens the door for the subsequent generations of cryptocurrencies to move in. Projects like Emogi, which started off with a focus on low cost, fast payments that can be used on the global stage and can scale to meet the needs.

With global tensions reaching levels not seen in decades, it appears a shift in the global financial infrastructure is underway. China has announced that it’s Digital Yuan is near completion and will be released in the near future, and the Governor of the Bank of England recently discussed the possibility of replacing the US Dollar with a Digital Currency.

There is a growing push happening around the planet that is moving our financial system towards digital currency, blockchain-based system, and it doesn’t appear that Bitcoin and Ethereum could handle such a capacity in their current states.

It’s not like there aren’t people working on the problem. There are multiple firms and millions of dollars being directed towards scaling solutions for BTC and ETH, but they might not be ready when it is truly needed.

So for all the people out there pointing to Bitcoin dominance and saying it’s only going to increase, or that Ethereum’s established network effect is too much for another altcoin to overcome, just wait until a billion people come looking for an alternative to high fees and long confirmation times as they just want to use their crypto to pay for a meal.

As more and more of the world migrate to the crypto space, and the Internet of Things world continues to expand into every facet of our lives, projects like Emogi and IOST are well-positioned to meet the demands of the coming shift.
Serverless Full-Text Search with AWS Lambda and EFS
By: Arseny Yankovski
Tags: AWS, Scalability, Java, Open Source, Serverless

For a long time, I was looking for a serverless full-text search solution. Of course, there are managed solutions for this, like Elastic Cloud or Amazon Elasticsearch Service. But all these solutions have a fixed cost and do not scale automatically.

I also saw a couple of attempts to run Apache Lucene with Lambda and S3, but these had a couple of issues. Firstly, Lucene is a Java library, and JVM had terrible cold start issues. Well, not anymore with GraalVM, which generates a native executable out of a JAR, reducing cold start and improving overall performance. Secondly, S3 is a bit too slow. Guess what? You can use EFS with a Lambda now, and it has much better latency.

So I thought to myself, why not build a simple proof of concept and see how it goes.

Approach

I used the Quarkus framework because of its small footprint and the ease of creating a native executable. This will help me solving the cold start.

Reading a Lucene index is thread and process safe. It means that read throughput is only limited by the AWS Lambda and EFS service limits. Writing, however, is a different story. An IndexWriter needs to obtain an exclusive lock on an index and will fail if another process is holding it.
Scalability in Blockchain
By: Dan Grichevsky
Tags: Blockchain, Ethereum, Scalability, Bitcoin

Scalability is a big issue at the fundamental core of blockchain technology. How can millions of people use bitcoin to transfer funds between themselves if there is a limit on the number of transactions the network can process at any given time? The answer is they can, but true adoption of this payment system will never occur if the fees and processing time for these transactions is too high. This article will discuss some of the proposed scaling solutions for blockchain.

First Layer Solutions

First layer solutions propose to change a fundamental aspect of the cryptocurrency. For example, a first layer solution for bitcoin would be to increase the block size from 1MB to 10 MB or by reducing the block creation time from 10 minutes to 5 minutes. A first layer solution requires a hard-fork of the cryptocurrency which means the creation of a new cryptocurrency altogether. Bitcoin Cash hard-forked from Bitcoin after some developers wanted to increase the block size from 1MB to 8MB. For Ethereum, a proposed solution has been sharding which means breaking a blockchain up into manageable parts. The shards are responsible for different groups of transactions, which increases the throughput of the network. This solution is expected to be put in production in 2020 after the Ethereum network moves to a Proof-of-Stake (PoS) consensus algorithm.

Second-Layer Solutions (Off-Chain)

Second-Layer solutions involve adding a new protocol on top of the already existing one. Transactions are off-loaded onto a side-chain to relieve congestion on the main blockchain. The lightning network for Bitcoin is an example. The lightning network will instantiate smart-contracts on top of the Bitcoin protocol to allow for the creation of private off-chain blockchains that help facilitate the processes of the network. Two proposed second-layer solutions for Ethereum are the Raiden Network and Plasma. The Raiden Network allows users to create State Channels which broadcast information to the rest of the network. Plasma utilizes child chains that run off the main blockchain to process transactions and report the consensus back to the main chain. Each child chain can have its own set of rules and processes which allows them to target specific types of transactions.

Scalable Consensus Mechanism

Another proposed scalability solution is to change the consensus algorithm of the protocol. Consensus algorithms like PoS or the Byzantine Fault Tolerance (BFT) allow for a higher throughput of transactions per second and overall greater efficiency in the network. Hyperledger is an example of a project that utilizes BFT.

Currently, blockchain only supports two out of these three properties — scalability, security, or decentralization. By implementing the proposed solutions above, it will hopefully reach a point where scalability is solved and the focus of the blockchain can be attaining true decentralization for its network.
Load testing with Apache JMeter
By: Arie M. Prasetyo
Tags: Load Testing, Performance, Test, Scalability

Write a basic test

There are two ways we can use JMeter:

Graphical user interface Command line interface

It is recommended that we run actual test on CLI. But we can write a test & have a test-run on the GUI. Start the GUI by running this command:

$ apache-jmeter-5.4.1/bin/jmeter

Apache JMeter GUI

In this scenario, we are going to write a basic performance test targeting randomcolour.com, a very simple website.

I’ll rename the test plan as “Load testing randomcolour.com”.

Thread group

Next, let’s add a thread group on our test plan. Thread group determines user flow and how a user would behave. Each thread represent a user.

Add a thread group

In this thread group, I am adding 5 users (threads), ramp all users in 5 seconds (thus 1 second after each other). And I’ll only be running the test once (loop count = 1).

Thread configuration

Sampler

We want the thread to send an HTTP request. To do this we can add a “Sampler” to the thread group.

Add a sampler

In this sampler, I’m sending a request to randomcolour.com , using an http protocol.

Sampler configuration

As you can see, we can configure many parameters on this screen. We can simulate a POST request, simulate form filling action, file uploads, etc. But in this example I’m just going to request a simple HTTP request.

Timer

We can simulate the pause/delay when users click a website by adding a “Constant Timer”. Let’s add a timer to the HTTP request we’ve just created.

Add a timer

I’m going to use the default value of 300ms.

Listener

To see the result of our test on the GUI, we can add a “Listener”. In this example, I’m adding a “View Results Tree” listener to the thread group.

Add a listener

Run test on GUI

Remember, you should always run the test on JMeter CLI interface. But we can do a test-run on the plan we’ve just created. First we need to save it as a .jmx file. In this example, I name the file test-randomcolour.jmx .

To start a test, click the “Start” button on the top toolbar.

Here is the result of our test as shown in the “View Results Tree” screen.
Sharding is Here: What’s Next?
By: Apollo Fintech
Tags: Scalability, Apollo, Sharding, Blockchain, Bloat

by Marvin Dumont

Apollo Foundation’s revolutionary sharding protocol has arrived, making Apollo one of the first crypto ventures to successfully launch the feature on its network. It solves long-term challenges with blockchain such as bloat.

The first shard executes at block 2,250,000 on April 1. (See important update on sharding release.)

As that happens, these take place:

It will split the ledger database into time-based (or blockchain height-based) segments. The split will be made on each node in the consensus. (Each node “knows” when that split occurred.) Each shard will be completed every 750,000 blocks after 1,400 blocks of acknowledgment. Information will be added to each shard/segment that allows blockchain operations to happen without queries to the full ledger database. This will optimize the blockchain processing code to use the last segment for most operations. Make each segment “trusted” by signing its summary tables. Make full blockchain downloading optional/delayed. Download the latest segment of the ledger and start processing. Download previous blockchain segments on demand. Download blockchain from last to first blocks.

Bloat brings sustainability concerns. Over time, there will be billions of blocks (or more) on-chain. Although each block adds a tiny amount of data, the accumulated info will eventually grow to astronomical sizes.

Sharding makes downloading the full blockchain unnecessary. You get the latest slice of the ledger — and start processing. Previous slices can be accessed on demand — from last block to first.

The result is remarkable speed and scalability.

Here’s more info on Apollo sharding.

Sharding: A Solution To Blockchain Bloat

Explaining Sharding: A Game Changer For Blockchain

Sharding Will Launch April 1

Apollo (APL) all-in-one privacy currency combines features of mainstream cryptocurrencies in a truly private and unregulatable platform. With two-second block speed, APL is one of the fastest cryptos on Earth. The privacy platform lets “Apollonauts” transact and send data anonymously via Encrypted Messaging, Private Ledger, Decentralized Exchange, IP Masking 2.0 and Coin Mixing. Learn more at www.apollocurrency.com
What is Micro-Frontends(MFEs)?
By: Bhargav Shah
Tags: Frontend Development, Scalability, Techniques, Architectural Design, Micro Frontends

Cons:

Micro frontends are not perfect for every project. They are a solution for scaling development. If you only have a hand full of developers and communication is no issue the introduction of micro frontends won’t bring much value.

Increased Payload size

Environment differences

Operational and governance complexity

Works best for big teams (hundreds of developer)

No global standards of development

Implementation example 1:

Code at,

Implementation example 2:

Code at,

A JavaScript framework for front-end microservices:

Build micro frontends that coexist and can (but don’t need to) be written with their own framework. This allows you to:

Use multiple frameworks on the same page without refreshing the page (React, AngularJS, Angular, Ember, or whatever you’re using)

Write new code, possibly with a new framework, without rewriting your existing app

Lazy load code for improved initial load time.

Micro frontends are indeed quite popular in the e-commerce sector. In 2012 the Otto Group, a Germany based mail order company and one of the world’s largest e-commerce players started to split up its monolith. The Swedish furniture company IKEA and Zalando, one of Europes biggest fashion retailers, moved to this model. Thalia, a German bookstore chain, rebuilt its e-reader store into vertical slices to increase development speed.

But micro frontends are also used in other industries. Spotify organizes itself in autonomous end-to-end teams they call Squads. Canopy, an American tax software startup and the sports streaming service DAZN use micro frontends to build its applications.

Conclusion:

Microfrontends are sections of your UI, often consisting of dozens of components, that use frameworks like React, Vue, and Angular to render their components. Each microfrontend can be managed by a different team and may be implemented using its own framework. It is practical and suggested to use just one framework for all your microfrontends, although you may add additional frameworks when migrating or when experimenting.
You need a Flywheel
By: Paul Sturrock
Tags: Design Thinking, Scalability, Strategy, Business Strategy, Startup

Roger Martin is one of the most practical writers on strategy, because he is skeptical about most traditional approaches and he gives you actionable insights as he does . My favourite is his advice to focus on asking yourself “what would have to be true?” and doing reality checks instead of trying to predict the future.

Expenses are completely within a company’s control. A firm can decide exactly how much in the way of raw materials to purchase, how many people to hire, how many square feet of office space or manufacturing space they need; all of those are completely within the company’s control, so you can actually plan for these things. On the other hand, sales targets are entirely dependent upon customers. You can hope for future sales. You can say, ‘We want to hit one billion dollars in sales this year’, or $10 billion — as much as you want; but that is not going to make it happen.

Sales will come from making choices that compel customers to want to buy your products. Yet we never see any mention of that in any strategic plan.

…but often, when you look backwards, your actual strategy changed and shifted, based on emergent conditions in the marketplace. This is a great insight…Unfortunately, some strategic planners have taken it to mean that ‘You shouldn’t bother planning ahead’.

You can’t say, ‘We have done the analysis, and we have come up with the right answer’. What you can do is imagine possibilities and make choices that you believe to be the most compelling you can make. Sometimes you will be right, and sometimes you will be wrong. That is the nature of strategy, because that is the nature of life.

The challenge is to figure out ‘What 10 things must be true for this strategy to be a great one?’ Your answers might be things like, ‘Customers will have to behave this way; the distribution channel will have to value X; competitors will have to not do Y’, etc.
Enterprise-Scale Data Engineering with Python: Evaluation of Bodo with the TPCxBB Q26 Benchmark
By: Zhuchang Zhan
Tags: Predictive Analytics, Benchmark, Scalability, Big Data, Efficiency

TPCxBB Q26 Benchmark

The customer chose the industry-standard TPCxBB benchmark suite since it can represent data engineering workloads for various applications, including machine learning pipelines. It is designed to represent advanced analytics use-cases that utilize structured, semi-structured, and unstructured data. In our experience, even TPCxBB is too simple to represent complex data pipelines in ML applications, and therefore cannot fully exercise Bodo’s optimization capabilities. But it can serve as a good starting point for evaluation.

The TPCxBB benchmarks represent workloads in a retail setting with various data sources (such as online and in-store transactions, logs, and reviews). Still, the benchmark patterns apply to all verticals. In addition, the data generator is scalable and can generate data at any scale factor, which is very useful for scalability tests.

This evaluation focuses on Query 26 of the TPCxBB benchmark suite, which clusters retail customers into groups based on their purchase history. This particular benchmark is interesting since it represents data preparation and feature engineering stages of an ML pipeline well. Further, focusing on a single benchmark allows examining many factors (memory consumption, energy cost, etc.) in more depth. Below are the Spark and Bodo versions of the code. The Bodo version uses native Python/Pandas APIs and is a straightforward translation of the original SQL code. The Spark version uses SQL for faster speed but for simplicity could also be written in the equivalent PySpark DataFrames APIs which are wrappers around the SparkSQL engine.
Geosharded Recommendations Part 3: Consistency
By: Devin Thomson
Tags: Elasticsearch, Scalability, Kafka, Distributed Systems, Recommendations

Datastore Consistency

Elasticsearch is classified as a near real-time search engine. What this means in practice is that writes are queued into an in-memory buffer (and a transaction log for error recovery) before being “refreshed” to a segment on the filesystem cache and becoming searchable. The segment will eventually be “flushed” to disk and stored permanently, but it’s not necessary to be searchable. See this page for details.

Figure 1 (left), Figure 2 (right)

In Figure 1, the write has been added to the in-memory buffer but is not yet searchable.

In Figure 2, the in-memory buffer has been refreshed as a new segment in the filesystem cache, now searchable.

The state is inconsistent between the mapping datastore and search index, and the document will remain in geoshard B.

The solution to this is using a workflow that guarantees strong consistency within search index. The most natural API for moving a document from index to index is the Reindex API, however that relies on the same realtime search expectation and is thus unacceptable.

Elasticsearch does provide the Get API, however, which by default comes with functionality that will refresh the index if attempting to fetch a document that has a pending write that has yet to be refreshed.

Using a GET api that refreshes the index if there are pending writes for the document being fetched eliminates the consistency issue. A slight increase in application code to perform a GET + Index rather than just a Reindex is well worth the trouble avoided.

A final note — the mapping datastore may also have an eventually consistent data model. If this is the case then the same considerations must also be taken (ensure strongly consistent reads), else the mapping may point to the document being in a different geoshard than it actually is in, resulting in failed future writes.
Vertical scalability Vs Horizontal scalability
By: Sofiene Ben Khemis
Tags: Scalability, Cloud Computing, Software Development, DevOps

It’s better to have infinite scalability and not need it, than to need infinite scalability and not have it. Andrew Clay Shafer

As the server load increases, and it’s time to add hardware resources to your infrastructure, you have two approaches: increase your resources vertically or horizontally.

Vertical scalability

Vertical scalability is the most intuitive, it means adding resources to a single server (by adding RAM, boosting its CPU …). It’s simple but the cost of material capacity is rapidly exponential.

This solution has limits, You can probably multiply your server’s capabilities by 2, maybe even by 10, but not by 100.

Also usually, vertical scalability requires downtime while new resources are being added.

But this remains in most cases the most pragmatic solution, and in particular for applications in production for many years.

Horizontal scalability

Horizontal scalability is like adding new servers doing the same type of task.

This allows you to use only standard servers with commodity hardware. But the software implications are quickly important.

The application must be stateless which means that the data is stored in another service (in your database, in a distributed cache…). It is ideal, You should design your development with this goal.

Thanks to this, in case of hardware failure on one instance, other instances will be able to absorb the load with a minimum service interruption for the user.

Conclusion

Overall, Vertical scalability involves using a computer that offers many possibilities for adding parts, on which it is possible to put a large amount of memory, many processors, several motherboards and many hard drives.

On the other hand, Horizontal scalability involves adding computers to cope with increased demand for a service. The most common method is load allocation by the use of a cluster of servers.
Using Change Streams in MongoDB
By: Rajdeep Mandal
Tags: Scalability, Change Streams, Mongodb, Event Listener, Database Design

Using Change Streams in MongoDB

In the pursuit of making scalable systems, I have discovered that using listeners in your database makes your life many times easier. Here are some of the instances, when these listeners saved my life. Well, saved is a bit of an overstatement, but definitely made it more comfortable.

The building of reactive systems

Data pipelines to other systems

Updating parameters on UI and live notifications

In a system where there is a frequent influx of data, which eventually gets dumped in a database, there are two alternate ways of tackling this. One way to do this is to tail the oplogs. The other way is to incorporate this logic somewhere before interaction with the database. I am not a fan of either, as these are what you call dirty ways of programming.

An Illustration of Having a Different Server for Listeners

The idea here is to build an independent application, that subscribes to a particular category of DB changes, and immediately reacts to them.

If you are already decently aware of what Change Streams are, you may directly jump to Change Stream Output for detailed intricacies.
A Newbie’s Chronicles on Distributed Data Processing — S01E03: Grand Finale
By: Athitya Kumar
Tags: Scalability, Experience, Data, Distributed Systems, Learning

Roadblock: The DAG-ger that cut deep

The application logs by themselves weren’t very useful to debug as we could just see tasks failing — but not the exact conditions in which they were failing. For example, we saw that our code was still failing due to a memory issue at the JOIN step. Which was weird — as memory shouldn’t have been an issue after the optimization changes we had made in the previous episode.

To figure that out, we turned to our secret arsenal: the Spark Web Console!

Sample Spark Web UI, credits: https://spark.apache.org/docs/3.0.0-preview/web-ui.html

Basically, this is a Dashboard UI provided by Spark, that throws a lot of useful hints such as the stages, tasks, time taken, nodes status, memory metrics, etc in a step-by-step manner for a job on-the-fly right when it’s running.

It came in clutch like a Swiss-knife for debugging, to say the least! As we looked at the current application’s run, we saw that for some reason, one of our earlier pre-processing steps to read incremental data was being called again during the JOIN operation, instead of re-using the one we had already got as output for in the count() operation stages.

The mysterious case of re-reading incremental data

Thankfully, Sanket was familiar with this issue as he had faced this before himself once and he explained it to us in detail.

Basically, Spark keeps versions of the computed values according to Catalyst’s Logical Plan, in a DAG (Directed Acyclic Graph). Now, depending on the memory availability (or requirement) for its next operation, the earliest values stored in the DAG keep getting purged one by one till we’ve made enough memory for the next operation to happen.

Which means, one of the variants of our incremental DataFrame must have gotten purged from the DAG to accommodate the join operations, and it was trying to read them all again and start processing from the beginning. This is why it had hit a memory issue — and the job started failing after a maximum tolerance of 4 retries for the failing task.

And here I was — thinking that if the DataFrame was in memory while successfully executing the count(), it’d still be in memory for the consequent JOIN steps. The very fact that the DataFrame could get purged from the DAG in between these 2 lines of code — felt like being betrayed by my own code. 😢

Trust issues: “Hello there, long time no see.”
A Newbie’s Chronicles on Distributed Data Processing — S01E02: Optimizations Galore!
By: Athitya Kumar
Tags: Scalability, Experience, Data, Distributed Systems, Learning

Optimizations, Optimizations & Optimizations

It’s usually tough to find out scopes of optimization in Spark-esque code, especially if you’re new to it — however, the trick is to pay attention to the important aspects like Memory Optimizations & Minimizing Data Shuffling.

So, we called upon one of the experts in our team to suggest scopes in improvement. Enter, Sanket Sahu. Within an hour of setting the context and running him through the POC code, he had a couple of suggestions that apply to Spark jobs in general:

Use of better memory configs:

The memory configs usually follow a bell-curve. If you provide fewer executors and more memory per executor, data shuffling is minimal; but the garbage collection becomes an overhead and parallelism reduces. On the other hand, if you provide more executors and less memory, though you utilise parallelism, there would be a lot of data shuffling as each executor has lesser memory.

It’s about finding a balance/trade-off between these 2 cases!

Minimize data shuffling:

Almost all of the heavy-lifting takes place in the executors, and very minimal auxiliary steps run on the driver. Due to this distributed nature, there’s a lot of scope for data shuffling: ie, chunks of data being sent to-and-fro within the cluster.

Wait, hold up. I got lost. Driver, Executor & Shuffling?

Oukay. Sorry, I got ahead of myself. In a nutshell:

Driver: Process that mainly handles reading our code, splitting it into stages/tasks, splitting and delegating them onto executors, etc. Somewhat similar to a dedicated Scrum Master.

Executor: As the name suggests, executors are processes that run in the Data/Worker nodes — they work on the tasks delegated to them by the Driver. Which is probably analogous to the folks picking up tickets in a sprint.

The Driver-Executor Architecture ft. Cluster Manager

Data shuffling: Now we know that chunks of DataFrames are stored in the Distributed File System across the cluster. So, any time we use an action like df1.join(df2), chunks of df1 and df2 have to be joined first and then all these resultant chunks have to be logically merged to give the final result. To facilitate this, data needs to flow across the cluster.

Sooooo why is Data Shuffling bad again?

Hmm, with the usual scale of data we have to process with Spark jobs, unnecessary data shuffling would result in unnecessary network calls within the cluster to make the data flow. As this may indeed end affect the performance, it’s usually desirable to have the least amount of data shuffle.

For example, let’s consider the JOIN operations we were executing:

The JOIN operations from our POC

To remove existing DELETE & UPSERT entries from our existing data, there’s no need to join entire DataFrames. It’s very much enough if we just join the column(s) that’s considered as the primary key(s) — and then we can just retain those non-matching primary keys from our existing data.

In similar lines, the optimized changes would look something like this:
A Newbie’s Chronicles on Distributed Data Processing — S01E01: Pilot
By: Athitya Kumar
Tags: Scalability, Experience, Data, Distributed Systems, Learning

The Required Capability

Now, let me explain the capability we required and recently added support for. As the Data Engineering Org, we act as a bridge between the source teams that have the ready consumable data, and the analysts who want to run performant queries on such a huge scale of data to derive insights.

In this particular case, we had to support around 16TB of incremental data on a daily basis from the source team, and make it available to the analysts by the next day. To give a better picture, let’s say our source team (say, Nick Fury) has Avengers data — that the analysts (say, SHIELD scientists) would like to derive insights from.

We have some existing data in one of our hive tables, and we keep receiving incremental data every day — we need to merge this incremental data into our hive. Which might look something like this:

Historic Data — “The Original Avengers”. Phew, good times!

And incremental events flowing in, look something like this:

UPSERTs — ie, Updates + Inserts

Incremental Updates + Inserts — upto “Avengers: Endgame” :)

DELETEs:

Incremental Deletes — also due to “Avengers: Endgame” (sob)

The expected outcome? After merging the incremental data with our existing historic data, we need to store this materialized data back into the same Hive table:

Final Materialized Data, after applying Inserts/Updates/Deletes — ie, after the end of “Avengers: Endgame”. Also becomes the Historic Data for the merging next time!

Of course, the exact order does not matter — as there’s no concept of primary keys in Hive, and data might indeed get shuffled during our processing. However, the exact snapshot of the final data needs to match row by row.

The main constraint of this capability is not the merging itself, but rather making it scale for such a huge amount of data: ~16TB/day!
Horizontal Scaling Vs Vertical Scaling
By: Mayur Agarwal
Tags: Scaling, Databse, Scalability, System Design Interview

Difference between them-

Load Balancing —

In horizontal scaling, when there too many requests it’s important to direct those requests to different machines, so that the load on any single machines isn’t too high, otherwise there’ll be no point of horizontal scaling if we kept directing every request to the same machine.

In vertical scaling, there’s no need of load balancing as there we haven’t changed the number of systems, we’ve optimized or increased the processing capacity of the system.

Single Point of failure -

In case of horizontal scaling, there’s no single point of failure because there is more than one machine, if one of the server crashes than, the requests will be redirected to the other servers. So it doesn’t affect too much. There requests that were made to the crashed server will now be made on other servers.

In case of vertical Scaling, there’s only one machine, so if the server crashes then, there is no other machines where the requests can be made, so this is the single point of failure. Whole process stops if the server crashes.

Database —

In a database world, horizontal scaling is usually based on the partitioning of data (each node only contains part of the data).

In vertical scaling, the whole database resides on a single node, and the scaling is done through spreading the load between the CPU and RAM of the machine.

Network Calls —

In case of horizontal scaling, all the network calls between the servers (Remote procedure calls) will be over the network, and network calls are slow.

In case of vertical scaling, we have inter-process communication (a mechanism that allows processes to communicate with each other and synchronize their actions) and that’s fast.

Hardware limit-

In case of horizontal scaling, upgrades are easier, all you need to do is add an additional node. You can say that, an infinite number of nodes can be added.

In case of vertical scaling, there is limited scope for upgrades. A server can only be so big. What happens when your database can no longer fit on the largest available server?

Complexity —

In case of horizontal scaling, there is massive increase in complexity, as we now have other additional things to focus on, like load balancing, network calls between different servers, etc. So the chances of bugs also increase. So management of overall infrastructure becomes more complex.

In case of vertical scaling, it is easier to implement and administer , there’s a single server to manage, and that’s it. So we just have to focus on maintaining a single server.

Examples —

For horizontal scaling — MonogoDB, Cassandra and Google Cloud Spanner.

For vertical scaling — MySQL, AmazonRDS.
Scaling with Nginx; Cost-Performance Analysis
By: Krishnakanth Alagiri
Tags: Scalability, Docker, Kubernetes, Nginx

Originally Posted on thekrishna.in

When you realize your system is getting slow and is unable to handle the current number of requests even with optimizations, you need to scale the system sooner than you can optimize further. Building a scalable system also drives to a lower Total Cost of Ownership (TCO). Proper scaling in process-intensive applications embraces interesting new scenarios, notably in data analytics and machine learning. Traditionally, you have two options, Horizontal scaling, and Vertical scaling.

Pokémon Go — A Successful Scaling Story

If you downloaded the Pokémon GO app right at its launch, you might have faced several issues on server unavailability for some minutes. Based on the hype around the announcement of Pokémon GO, one might have already presumed what chaos was going on in the backend infrastructure. Pokémon Go engineers never imagined their user base would grow exponentially, exceeding the expectations within a short time. With 500+ million downloads and 20+ million daily active users, the actual traffic was 50 times more than their initial expected traffic.

This makes Pokémon GO, one of the most exciting examples of container-based development and scaling in the wild. The logic for the game runs on Google Container Engine (GKE) powered by Kubernetes. Niantic picked GKE for its capability to orchestrate their container cluster at large-scale, saving its focus on deploying live changes for their players. So by provisioning tens of thousands of cores for Niantic’s Container Engine cluster, the game brought joy into all the Pokémon Trainers around the globe.

What is Horizontal scaling?

Horizontal scaling implies that you scale a cluster by attaching more machines or nodes into your pool of resources. Scaling horizontally is like thousands of minions will do the work together for you. Increasing the number of servers present in a system is the highly used solution is in the tech industry. This will eventually decrease the load in each node while providing redundancy and flexibility, thus reducing the risks of downtimes. If you need to scale the system, just add another server, and you are done.
Investing in Near Protocol
By: Vanessa Pestritto
Tags: Scalability, Investment, Decentralized Apps, Sharding, Blockchain

Investing in Near Protocol

Near Protocol is a sharded, proof-of-stake public blockchain, built by a world-class team that has developed some of the world’s only sharded databases at scale. The Near team has worked extensively to understand the problems developers face when trying to access new platforms as well as the usability of layer one protocols. Developers oftentimes are unable to leverage blockchain technology due to slow settlement times and scalability challenges. We at Xpring believe in empowering developers and providing the tools to address the pain of adopting and using blockchain technology.

Near has created a developer focused experience for building decentralized applications. They address scalability using sharding and their consensus model, Nightshade. The developer suite includes tools, resources, education materials, and the Near Studio. The studio is a space to create projects and write smart contracts in accessible languages, including TypeScript and Rust, and will eventually support all of the target languages that compile down to WASM.

I am excited to announce our investment in Near Protocol, alongside investors Accomplice, Electric Capital, Pantera Capital, MetaStable, Amplify Partners, Scalar Capital, Coinbase Ventures, Ideo, a_capital and others.

To build a decentralized application on near, apply for their beta program here. Near is hiring across all functions, see openings here.
I’m intrigued but a little confused.
By: Sarah Mock
Tags: Small Farming, Agtech, Scalability, Sustainability

I’m intrigued but a little confused. Do you mean in the way that agchem is often considered unsustainable in a way that drones are software are not?

It’s a really interesting point that I want to write more about, because actually I think a lot of real purists do think technology (namely tech tools- your drones/robots, sensors and automatons, etc, not to mention chemicals or genomics) IS unsustainable and feel negatively towards it. There’s definitely a group out there that thinks bringing tech into Ag makes it less sustainable because so few farmers can afford high tech tools and because those tools often entrench existing unsustainable practices (ie drip irrigation is high tech and more sustainable then flooding, but some would argue any kind of irrigation is unsustainable).

And for a lot of small farmers this position makes sense. A $2000 drone or $10,000/yr software subscription is not made for a 10 acre truck farm nor is it useful there. Bootstrapping and ignoring flashy new tech makes sense at that scale. But some people assume that because that is true at that scale, that tech at all levels is bad.

It’s say that’s where the biggest divide within agtech lies.
Tales of War — A Letter from the Front Line
By: Sever Moldovean
Tags: Scalability, Testnet, Sharding, Community, Blockchain

We just moved from the Build Phase to the Challenge Phase. After they had us dig the trenches and switch to different shards, it’s time to prove just how much we can process. So far, we worked together as a team, and did over 1 billion transactions, sometimes even 30.000 per second, before the slowest of us fail and then we start again. Today we are at 1.000 TPS and holding, let’s see what they throw at us next.

As we get better at configuring the nodes and letting them do battle, there’s more time to read and do some work for missions. They made us send our favorite food as part of a transaction — I missed you so much when I told the boys about your Orange Ginger Honey-Cakes.

Then they had us send transactions, with all the money that we earned or from all the different shards. I now understand which address is from where and why is that important and order of times faster.

The next one was more simple, we just had to fill the data field with a certain number of characters, to see how gas and storage costs works. I wrote the names we discussed for our son a thousand times, and still can’t decide which fits him best. Have you thought about it?

The toughest mission so far challenged the trust we have in one another and had us doubt our comrades. They wanted to re-enact the Byzantine Generals problem and have us risk it all on trusting just a few of us. But we pulled through, thanks to our field medic and his heavy lifting robot.

I hear they want us to get rid of all our Windows nodes. Something about the fastest Virtual Machine to be invented yet using libraries not yet fit for the OS I use. My friends are teaching me all about running Virtual Machines on top of Windows so I can deploy my nodes on Linux without too much pain. I think I like this Linux stuff, it was a bit hard at first, but looking back, I wouldn’t dare running our nodes on Windows once Battle of Nodes is over and the Elrond mainnet comes.
Elrond to Provide the Blockchain Component for the TailPath Supply Chain Management Ecosystem
By: Lucian Mincu
Tags: Supply Chain, Scalability, Elrondbd, Dapps, Blockchain

Their flagship product TailPath has already been successfully used in NATO military exercises, making it one of the world’s first blockchain use cases deployed for military purposes.

Elrond’s blockchain infrastructure will be used to anchor supply chain data via the TailPath mobile app, with fast transaction times enabling real time reading and writing of data at negligible fees. Through a development grant offered by Elrond Inc, SupplyBlockchain will take possession of ERD tokens to cover the initial smart contract deployments and transaction fees for integration testing and customer acquisition.

“Blockchain technology is a perfect fit to address the inefficiencies of supply chains. We are pleased to support TailPath integrate their product with Elrond in a native manner, without adding unnecessary complexity via secondary tokens. This will keep the costs for their customers as low as possible while also providing a meaningful use for our ERD token and our overall infrastructure.” concluded Beniamin Mincu, Elrond CEO.

About SupplyBlockchain, Inc

SupplyBlockchain is an internet startup tackling inefficiencies in supply chain management using blockchain and other technology innovations. It was established in 2018 in the US and has a growing portfolio of customers in the transport, logistics and distribution industries, who already enjoy substantial savings and loss reduction. Among its many innovations, It provided one of the world’s first blockchain deployment for the military. SupplyBlockchain focuses on achieving growth in the US markets via its TailPath ecosystem that include iOS and Android apps.

About Elrond

Elrond is a new blockchain architecture, designed from scratch to bring a 1000-fold cumulative improvement in throughput and execution speed. To achieve this, Elrond introduces two key innovations: a novel Adaptive State Sharding mechanism, and a Secure Proof of Stake (PoS) algorithm, enabling linear scalability with a fast, efficient, and secure consensus mechanism. Thus, Elrond can process upwards of 10,000 transactions per second (TPS), with 5-second latency, and negligible cost, attempting to become the backbone of a permissionless, borderless, globally accessible internet economy.
How To Store Your AWS Lambda Secrets
By: Timothy Jones
Tags: Security, AWS, Scalability, Lambda, Software Development

How To Store Your AWS Lambda Secrets

Don’t tell anyone your secrets. Do tell people how to store secrets. (Photo by Kristina Flour on Unsplash)

Most Lambda deployments use environment variables to pass configuration to the deployed function. It’s the recommendation that AWS makes in the documentation, and it’s an attractive choice because it’s easy, encrypted at rest, and allows for flexibility in how you get the values there in the first place.

There are many articles with good recommendations about lambda configurations already. Why should you read this one?

Instead of comparing and contrasting approaches, this is a how-to guide for anyone whose primary values are minimising cost without compromising scalability or security. If you have additional or different needs, I recommend reading the links above as well.

This guide is aimed at small to medium teams working in contexts where security matters, but fine-grained permission management might not.

Just tell me the answer

If you’re here from Google and just want a recommendation, feel free to skip to the end for the summary. If you want the detailed rationale behind the recommendation, read on.