fabiochiu/medium-articles huggingface dataset:
Lily Kneale welcomed to the team, and back to the island, as Marketing Officer
By: Charterhouse Lombard
Tags: Financial Services, Marketing, Charterhouse Lombard, Tcsp, New Hire

Lily Kneale welcomed to the team, and back to the island, as Marketing Officer Charterhouse Lombard Sep 10·2 min read

September has long marked a month of change, new beginnings and fresh stationary and this month is no different for Lily. Appointed by Charterhouse Lombard Limited as the new marketing officer in February, Lily is finally back on island and in the office, ready to take Charterhouse forward.

Lily thrilled to be at work

After doing her GCSEs and A Levels at Ramsey Grammar School, (a Manx girl through and through), Lily took a five year break to enjoy the big city life in Edinburgh and a brief stint of sunny island life in Gran Canaria, where she studied Spanish and French with Tourism management, before doing a masters in marketing back in Scotland.

Following a variety of part-time, student style jobs in everything from child care, driving Sainsbury’s vans, to a placement with the Department for Enterprise, she joins the Charterhouse team in Ramsey with an enthusiasm for “adult life”, the famed Isle of Man work-life balance, and a succulent that travelled all the way from Scotland.

On being back on island and joining Charterhouse she says:

“I am so excited to get stuck in, to learn all things finance and put into practice everything I’ve been at university for. It’s nice to change pace too, from one pretty but chilly island to another.”

Jonathan Leece, a Charterhouse Director, says:

“It is great to finally have Lily in the office working hard to bring together the marketing and Charterhouse’s plans for the future.”

We are looking forward to seeing what Lily will bring to the company and the office, and helping her to grow her expertise in financial services.

If you want to see what Lily (she/her) will be up to over the coming months for Charterhouse, make sure you follow her and Charterhouse on LinkedIn, and check out the website. Please direct all enquiries here.
This is Not Magical Realism, It’s a Memoir!
Tags: Mental Health, Mars, Magical Realism, Pain, Memoir

Can you feel the Red?

I was born on Mars, the red planet, yet all my childhood memories have no colour. Except for one maybe, where I am covered in dirt, my paws/hands, are green, and every colour visible to the human eye is around me. I am not too sure where it is or what it is but it has colours.

Oh, colours remind me, I think the number 3 gots to keep his anger in check, it is red after all. Let me give you some context. We Martians see numbers, alphabets in colours(many more than imaginable to the human mind); they also have personalities, I mean, I think 10 is too old to be doling out advice to the young kids out there. Don’t even get me started on days, months and years. Time, unlike Earth, is not linear on Mars and is distributed all around like a fluid in a vacuum. I would say, Time (yes, it’s a proper noun), is not linear on Earth either but humans are so scared of chaos and uncertainly, they all pretend that it is.

Why do humans find silhouettes beautiful?

Me?

Well, I am in-between spaces and time. Having left Mars a few years ago to be on Earth, I belong everywhere, every time and nowhere.

Why did I come to Earth?

You know, the charm of the big planet. With some great humans, stories, battles and of course cafes! I gave into the Neo-liberal fantasy.

Wait…

My best friend from my childhood

I can almost hear you getting confused. What the hell is this? What is this writing? Magical realism?

No. It’s not magical realism.

Why?

On Mars, this is just realism. I am telling you about my life. It’s a memoir of sorts. Is there magic in it?

You decide! You decide for how long you are willing to suspend your disbelief.

For how long are you willing to believe or not believe the world around you?

Who am I?

I am a Martian writer with a fake Human Name and this is my Memoir.

Image credits- My human love - Rishab
UNIQUE: UNTOLD SECRET Tales of Marsiada: ebook for kids and adults, This story is about believing in yourself, flight to space, the hidden cities on Mars, friendship with aliens
By: Ab R
Tags: Ebook For Kids And Adults, Friendship, Corona Virus Outbreak, Mars

UNIQUE: UNTOLD SECRET Tales of Marsiada: ebook for kids and adults, This story is about believing in yourself, flight to space, the hidden cities on Mars, friendship with aliens AB R Apr 23, 2020·2 min read

All things are possible to those who believe!Have you ever dreamed of flying into space? The heroes of this tale never thought that one day they would go on an important mission to Mars.This story is about believing in yourself, a flight into space, the search for hidden cities on Mars, friendship with aliens, and parallel universes.Three kind, courageous, and brave animals (African, American, and Russian), lived ordinary lives until they met each other. They grew up in different countries and did not believe in themselves. But one day everything changed; now they are not afraid of any trials. Together, they are unstoppable! These friends became a strong team and discovered superpowers in themselves.The prototype of Alan Maka, one of the main heroes, is the inventor Elon Musk. Dreaming of colonizing Mars, he cannot get permission from the government of the U.S.A. to send volunteers there. Therefore he sends three unique talking animals to the Red Planet. Their task is to find a suitable environment for people to live. However, what they discover there will completely change mankind’s ideas about Mars and advance science! But can they tell the world about their discoveries? What will happen on their way home..?Fall in love with astronomy! Fantasize about adventure and what can happen to you while traveling around the Galaxy! If you are attracted to everything unusual and mysterious, then welcome to the world of “UNIQUE”!

This Book is available now on Amazon
Challenge of Habitats Pt.1 — (A.I. Dev. Series)
By: Hriday Kharpude
Tags: Space, Mars, Artificial Intelligence, Machine Learning, Life On Mars

Intro

Hi All, I know it’s been a while since my last post, but let’s just cut to the chase… Have you guys ever imagined or even wondered about 3d printing? As in a machine on its own following a path going about printing material in a definite way creating a uniform structure. Well, then this post is for all those ingenious minds out there who, like me, visioning themselves at the peak of the A.I. industry, having contributed tremendously to the world.

It all started in NASA’s 3D-Printed Habitat Challenge. Just the way I felt, the title is not self-explanatory so you might just pass by thinking that it’s no big deal. Right? But here’s what it actually is…

NASA’s Challenge

As for outer space travel, if we just talk about Mars, it takes 6 months to get there in a short capsule-like compartment that might be smaller than the bathrooms in most of our houses. Once you get there, you are out of contact with Earth and have to patiently wait there for at least 2 years for the planetary positions to align themselves which is followed by an equally short trip of JUST 6 months to get back to Earth.

The Cabinet smaller than our bathrooms!!

Visit this page to know more!

Now a general mind would get fascinated or probably horror-struck by such atrocities, but a sharper mind would firstly think about how in the world are we going to stay in a capsule-like compartment for two years. So, if you caught along all of this explanation, let’s go back to the title NASA’s 3D-Printed Habitat Challenge. Now, pretty self-explanatory right!!! Even more so if you know that NASA has a Milestone laid in front of it which says :

“Year 2033 — Humans Landed on Mars”
Airdrop Extended to 06/16th
Tags: Fantom, Marstomars, Mars, Crypto, Projectmars

Airdrop Period Extended

Due to popular demand, we decide to extend Airdrop period to 2021/06/16th

As of right now, we’ve already distributed tokens to more than 500K+ Martians.

Rules

If you failed to participate in our Airdrop event, you can participate now by joining our discord channel and put your wallet address in the Airdrop Channel.

Official Discord: https://discord.com/invite/XeTZEjtbZC

Rule #1: For people who didn’t participate in the Airdrop event, write your address in our discord Airdrop channel, and you will get 500 MARS.

Rule #2: If you already claimed your airdrop successfully, you won’t get more tokens. Our server already has a record of you.

Rule #3: You should only put 1 address in the channel. If we detect you have multiple records, you will be counted as an abuser, and won’t get any tokens.

Rule #4: For people who participated in the Airdrop event and didn’t get any tokens, please put your wallet address there. It’s possible that our machine learning algorithm picked you as an abuser.

Airdrop will be distributed on the 17th!

Official Discord: https://discord.com/invite/XeTZEjtbZC

Official Website: https://www.projectmars.finance/

Official Telegram: https://t.me/projectmars_ftm

Official Twitter: https://twitter.com/ProjectMarsFTM
Month one is a wrap!! #WenHop
By: Mars Economy
Tags: Entrepreneurship, Mars, Cryptocurrency, Defi

We want to give a huge shout out to our early community who have been here through our first month supporting the launch of the Mars Economy protocol.

With everything that has happened in the past weeks, it’s crazy to think that it’s already been one month since we went live on the Liquifi exchange! We’re happy to hear the positive community feedback on both the farming capabilities and the decreased slippage on the Liquifi platform, we could not be more happy with our partnership and our decision to launch first on their exchange!

Shortly after the Decentralized Mars Token (DMT) launch, the Mars Prediction Market went live on the mainnet. Already over 100 predictions have been placed as our community members look to the future of Mars exploration and the projects that will get us there. This was a big step for us in helping spread knowledge around missions to the Red Planet which is a key part of our mission; this prediction market is one of the ways we’re doing that.

Our next milestone was the token launch on the PancakeSwap V2! It is one of the most popular decentralized exchanges and helped make the token more accessible to the public and those outside of the immediate community!

Out of everything that we have been able to accomplish in month one though, we are most proud of our community growth. In four short weeks we were able to grow from a community of 15 to well over 30,000. Humanity has always answered the call to explore new worlds, and we see this echoed in our rapid community growth as we ask the public to bring life to Mars!!

So what’s next? In the immediate future we will be launching the token governance side of the project, off-chain at first via Snapshot, and shortly after a dashboard for on-chain governance will be available. This is a priority for us as we look to the future, and hand this project entirely over to the community.

We are also working on ways to both stake and farm our native token, and will be moving the prediction market from its root in BUSD to our token, DMT.

Most exciting is our upcoming donation to a non-profit project focused on the expansion of humanity to Mars. Once our governance functionality is set, we will welcome our community to the first vote on tokens to be donated to colonization efforts!!

If you are a member of the community reading this we want to thank you for your efforts you have made along the way on the first legs of this journey. We would not be here without your support. If you are new to the project and reading about us for the first time, welcome! Feel free to drop in the telegram chat and meet the community — bring your Mars related questions or your meme of the day, we welcome all!

For more news and information on the project check out our social links below, and head to the Mars Prediction Market to place your first bet! Let’s bring life to Mars.

Website: Mars Economy

Twitter: MarsEconomy (@MarsEconomy) / Twitter

Telegram: https://t.me/marseconomychat

Instagram: MarsEconomy (@mars_economy) • Instagram photos and videos

Reddit: MarsEconomy (reddit.com)
.: Too Much Power :.
By: Adam Sommer
Tags: Astrology, Mars, Pluto

On Mars & Pluto

Art by Angela Alverez

The most basic definition of power is “the capacity to do something.” Vague, yet straight to the point. From the moment we scream into this world, to the moment we cringe and smile out of it, we are searching for power. Yet, the powers that be, will always be testing our ability to get any more power than we already have. Not just political and financial powers, but also the powers of nature. The further we step into the wild, the less power we realize we have. Same is true with the machine. The more we meld with it, the less power we seem to possess. Everything exists (or doesn’t) because of power.

Alice Walker commented on this powerful predicament we are all in by saying, “The most common way people give up their power is by thinking they don’t have any.” This is the trick. When we remember we have the power to change our minds, to do something about our actions, we come back to the center of the dynamic. We always have the power to choose wisely and how we respond to life, no matter the circumstances. It’s powerful to remember this.

Tomorrow (June 5th), two gods face off who have everything to do with power: Mars will oppose Pluto (Cancer/Capricorn). They are the co-rulers of Scorpio and whenever they align, we can expect much more prickles than goo. It is a macabre scene they operate in. Pluto lording over the underneath, Mars punching the walls up above. And when they oppose one another, the tension is close to a snap. It’s our job to let go before that happens.

We don’t have enough fingers to point to all the reasons we may feel powerless in 2021, and it’s probably for the best. Pointing or blaming is an odious activity. So, if you are still in lockdown, fuming and counting the days on the wall, what have you been putting off? There is something, no doubt that needs your power. If you are sick or in the throws of tremendous grief, how can you see it as a teacher? There is powerful praise to be found in there. And if you are swearing more than usual because of Elon’s tweets or eagerly waiting for the next reveal about Mr. Gates, your power is simply being given away. Take your power back. What can you actually change? Right now! HINT: It’s usually your something in your thinking…

If you feel Pluto and Mars are evil, you need a more powerful introduction. It won’t be easy, yet on the other side you will understand. Some would call this initiation.

Praise to Mars & Pluto

The Lords of shadows and tough content

The ones who show us our power once again

~*~

Eudaimonia,

Adam

kosmognosis.medium.com/follow

www.Patreon.com/adamsommer

….to support the creation of my writings/podcasts and be showered with gifts as well

www.Holestoheavens.com

…to adventure deeper into kosmos, mythos, and psyche
Oxygen Produced On Mars: How And Why?
By: Aryan Pandey
Tags: Space, How, Mars, Why, Science

After the lift-off of the Ingenuity helicopter from the Martian surface, NASA’s Perseverance mission reached yet another milestone. a device aboard the Perseverance rover was able to produce oxygen from the thin Martian atmosphere for the first time. This holds out new promise for future cr e wed missions that can rely on this technology for astronauts to breathe.

A device called the Mars Oxygen In- Situ Resource Utilization Experiment(MIXIE) Produced around 5 grams of oxygen, enough for an astronaut to breathe for 10 minutes.

The Procedure

Carbon dioxide makes up about 90% of Mars’s atmosphere. Oxygen makes up only 0.13% compared to 21% in Earth’s atmosphere. MOXIE, which inhales carbon dioxide and exhales oxygen, is designed to generate up to 12 grams oxygen per hour — about the same as the large tree produces. Placed inside the rover, it covers the size of a car battery, weighing 17.1 kg on Earth and 6.41 kg on Mars.

MOXIE first takes in Martian carbon dioxide and filters it of contaminants. It then uses an electrolysis technology to separate oxygen atoms from the carbon dioxide molecules at a temperature of around 800 degree Celsius. The oxygen is further isolated by a hot, charged ceramic component, which causes oxygen ions to merge into O2. Carbon monoxide is a byproduct, which is expelled back into the atmosphere.

A team on Earth will now analyse the purity of oxygen produced, said the Massachusetts Institute of Technology, whose labs are collaborating with NASA in the MOXIE project. MOXIE is only a test model. Further oxygen generators need to be about 100 times larger to support human missions on Mars.

The significance

Oxygen supply on Mars is essential for crewed missions- not just for astronauts to breathe but for rockets to use as fuel while coming back to Earth.

According to NASA, for four astronauts to take off from the Martian surface, a future mission would require around 7 tonnes of rocket fuel and 25 tonnes of oxygen — around the weight of a space shuttle. In contrast, astronauts living and working on Mars would require far less oxygen to breathe, maybe around one metric tonne for entire year. It would be an enormous challenge to haul the 25 metric tonnes from Earth to Mars for the return journey. This is where MOXIE’s role comes in.

NASA hopes to build a larger (1 — tonne) technological descendant of the experimental MOXIE that can do this job.
X21 Digital and MARS4 Partnership
By: Fred D
Tags: Nft, Ethereum, Btc, Mars, Cryptocurrency

The X21 Digital team is excited to announce the addition of Mars4 to our investment portfolio. Mars4 is a unique multiverse Interactive 3D virtual world staged on the Red Planet, Mars. It is the first virtual reality platform that leverages gamification through land ownership and exploration of the Red planet. For the first time, humanity can explore the four corners of the fourth planet virtually.

Like the top multiverse platforms in the decentralized gaming ecosystem, Mars4 plans to introduce a rich multiverse world while earning income through NFT land plots, furthermore through the MARS4 utility token ($MARS), token holders can terraform their NFT territory, buy certificates, avatars, logos, and other in-game assets within the virtual mars world.

Own a piece of Mars by grabbing your very own plot of land; however, one must act quickly as there is a limited amount of plots available for purchase, 99,888 Mars NFTs, to be exact. These NFTs have a significant use case within the virtual world of Mars4. Whenever a person buys a Mars4 NFT, they are credited with MARS tokens, which creates a unique passive income dynamic.

About mars4

Mars4 is a unique crypto multiverse. Staged in MARS, it offers a virtual reality platform offering users the unique opportunity of creating, monetizing content, and earning passive income through a first of its kind revenue-generating Mars NFT. These NFTs represent unique land plots that yield income from the in-game ecosystem.

About X21 Digital

X21 Digital is a Blockchain advisory and investment firm dedicated to helping startups accelerate their growth and exposure in the industry.

To date, X21 Digital has invested and advised several prestigious companies with exceptional teams giving the most optimal strategic advice and guidance to shortcut their path to success.

Useful Links

X21 Digital

Website: https://x21digital.com/

Twitter: https://twitter.com/X21_Digital

Medium: https://medium.com/x21-digital



Mars4

Website: https://www.mars4.me/

Twitter: https://twitter.com/MARS4_me

Telegram: http://t.me/mars4me_official

Medium: https://mars4-me.medium.com/
Announcement of Mars Migration Phase II Online Rate Update
By: Mars Project
Tags: Ethereum, Bitcoin, Mars, Pbtc35a, Crypto Mining

The Mars migration of Bitcoin mining rigs has had a breakthrough in the past few weeks with an incremental hash power of 57,959TH/s has been set up for mining to date. The corresponding mining outputs of wBTC will be paid out alongside the increase of pBTC35A online rate from 18.59% to 45.60%, taking effect from Singapore time (GMT+8) 23:59:59, September 7, 2021.

Stats of Hashrate Certificates as of Sep 7th

Besides, the team expected to have more batches of Bitcoin and Ether mining rigs delivered to sites over the next two to three weeks.

Till next time.

Mars Team

Sep 7, 2021

— — — — — — — — — — — — — — — — — — — —

Mars Project, the Standardized & Decentralized Hashrate Protocol

Website: https://mars.poolin.fi/#/

Learn More: https://mars.poolin.fi/#/about

— — — — — — — —

Mars Project Token Address

MARS (Governance Token)

https://etherscan.io/token/0x66C0DDEd8433c9EA86C8cf91237B14e10b4d70B7

pBTC35A (BTC Hashrate Certificate)

https://etherscan.io/token/0xA8b12Cc90AbF65191532a12bb5394A714A46d358

pETH18C (ETH Hashrate Certificate)

https://etherscan.io/token/0xA15690E9205De386Ce849889831C1668c300C1ad

— — — — — — — —

Trade & Add Liquidity

MARS (Token)

https://v2.info.uniswap.org/pair/0x64b91b92240bc1901855dd55ae632addb650d089

pBTC35A (Token)

https://v2.info.uniswap.org/pair/0x5b1e45ca08fa4d65aa7fdcf9e116990fb7fce73b

pETH18C (Token)

https://v2.info.uniswap.org/pair/0x2dc9d00da9542f91d5391be53ed1d58a04ec1c07

*Recommend 1inch for large volume transaction in case of malicious front running:

https://app.1inch.io/#/1/swap/USDT/pBTC35A

— — — — — — — —

Social Media & Community

Twitter: https://twitter.com/MarsProject2020

Telegram: https://t.me/marsproject1

Discord: https://discord.com/invite/hjHAvJpaVM

Medium: https://marsproject.medium.com/

WeChat (CN): Mars_Poolin2021
ACCESSING HADOOP FILESYSTEM API WITH PYSPARK
By: Somanath Sankaran
Tags: Pyspark, Hadoop, Big Data

This is one of my stories in spark deep dive

Spark is a parallel processing framework which sits on top of Hadoop filesystem.So there are few common use-case which requires handling hadoop filesystem.Let us the see the following in detail

Hadoop Filesystem API Accessing Hadoop file-system API with Pyspark Common Use- Case with Hadoop FileSystem API

Hadoop Filesystem API:

It is a abstract class in Java which can be used for accessing distributed filesystem.

Since it is a abstract class it has a get method which need the configuration of the filesystem and it return a static FileSystem class which will use to access hadoop filesytem and we will use this to do common operation like ls ,copyToLocalFile

Accessing Hadoop file-system API with Pyspark

In pyspark unlike in scala where we can import the java classes immediately.

In pyspark it is available under Py4j.java_gateway JVM View and is available under sc._jvm

Step1:import the abstract class

we can import the abstract class

Step2:Creating the static class

Before passing the hadoop conf we have to check if the spark integration to hadoop uri is made correctly

For example in my case this is not pointing to hadoop filesytem .So I will set this to hadoop filesystem (this is optional as in most prod systems it will be set)

Next step is to create the static class by passing this hadoop conf object

Common Use- Case with Hadoop FileSystem API

Let us see the following use case

Make Directory copyfile from local list hdfs directory

4.Copyto local

Make Directory

we can make dir with mkdirs



throws

public boolean mkdirs( Path f)throws IOException

We will import the Path class also from same jvm

And it is verified with hadoop fs -ls

copyfile from local

Since we need to copy from local file:/// is used

Checking file exists

list hdfs directory

we can use glob status to match all the dir with glob pattern as shown below.

this with return a java object and we can use list comprehension to get the attributes of file

Copyto local

check if it is copied to local

That’s all for the day !! :)

Github Link: https://github.com/SomanathSankaran/spark_medium/tree/master/spark_csv

Please post me with topics in spark which I have to cover and provide me with suggestion for improving my writing :)

Learn and let others Learn!!
Core PySpark: Inner Join on RDDs. Below you will find a simple example…
By: Alexander Gao
Tags: Apache Spark, Innerjoin, Pyspark, Data Science, Rdd

Let’s begin. First, we simply import pyspark and create a Spark Context.

Import PySpark

We are going to use the following very simple example RDDs: People and Transactions.

Create two RDDs that have columns in common that we wish to perform inner join over.

New let’s perform some data-formatting operations on the RDD to get it into a format that suits our goals. We will split each split each row using commas as a delimiter, and remove headers.

Suppose we want to perform a join on the Transactions and People dataset. The property that each dataset has in common is ‘name’.

If you consult the Pyspark documentation, performing a .join() operation on RDDs uses a (Key, Value) paradigm to find the intersection between sets. Therefore, prior to performing the join, we should format our datasets so that they conform to the (Key, Value) format required by Spark (conforms to the MapReduce paradigm).

Documentation: https://spark.apache.org/docs/latest/api/python/pyspark.html

Notice how we have formatted our RDDs as (Key, Value) pairs, where the Key is now the property we would like to perform the inner join on.

You can see the results of joining — since we formatted the RDDs as (Key, Value) pairs and then performed the inner join, the resulting RDD is the inner join on whatever we chose to be the Key. In this example, we now have a new RDD that combines the People RDD with the Transactions RDD to give us a table of transactions, each of which include additional information about the person responsible for that transaction.

I hope this was helpful to some of you!
Introduction and Installation of PySpark for Mac Users
By: Mrinalini M
Tags: Mac, Big Data, Apache Spark, Pyspark, Spark

This blog briefly introduces you to the not-so-new but one of the most widely used technologies in Big Data and AI. Apache Spark!

Yes, Spark is almost 10 years old by now. This blog also throws some light on PySpark and the steps required to install it in your machine.Let’s get started.

What is Spark? Why Spark at all?

Spark is an open source, big data platform which helps deal with larger datasets in a very fast and efficient way. It is a large scale data processing framework that performs cluster computing and processes tasks quickly, by splitting up the data among several nodes (think of it as computers) and performs computational task in each node of a cluster, in parallel. We usually use big data technologies when we have to deal with data which is so large that it doesn’t fit into your machine’s RAM. Many of you might be wondering why don’t we just go with Hadoop’s MapReduce to carry out such work. Well, that’s because Spark is 100x faster than Hadoop’s MapReduce. Here is how….

MapReduce writes the data to disk after each map and reduce operation while Spark keeps most of the data in memory after each transformation.

Be it a Data analyst or a Data Scientist who wants to rapidly query, analyse, transform or draw insights from the “Big” data, Spark is the go-to solution. Because spark has the libraries for SQL, Machine Learning, stream processing and graph computation. It also supports various programming languages like Java, Python, R.

PySpark

PySpark is an API for Spark framework released by Apache Spark to support Python with Spark. One can easily leverage all the capabilities of Spark using this API. Let’s look how we can setup PySpark in our machine.
PySpark Word Count
By: Gülcan Öğündür
Tags: Pyspark, Spark, Word Count, Python

Apache Spark is an open-source, distributed processing system used for big data workloads. PySpark is the API written in Python to support Apache Spark. If you have used Python and have knowledge about Pandas then it will be easier to learn PySpark. The goal of this post is running with PySpark and make a Word Count application.

You can find many articles on Medium how to install Spark. We will do just a practice on how we can use PySpark. After installing PySpark, we will write PySpark in a Jupyter Notebook.

There are two ways to have PySpark available in a Jupyter Notebook:

Configure PySpark driver to use Jupyter Notebook: running pyspark will automatically open a Jupyter Notebook

will automatically open a Jupyter Notebook Load a regular Jupyter Notebook and load PySpark using findSpark package

I prefer the second option which is more commonly used. In order for Python to find the Spark, download the “findspark” library and start it with findspark.init() function.

RDD (Resilient Distributed Dataset)

A RDD is a parallelized data structure that gets workload distributed across the worker nodes. They are the basic units of Spark programming.

In order to work with RDDs, we need to create a SparkContext.

A SparkContext is the entry gate for Spark environment. For every Sparkapp you need to create the SparkContext object. It allows your Spark Application to access Spark Cluster with the help of Resource Manager ( Spark Standalone/Yarn/Mesos).To create SparkContext, first SparkConf should be made. The SparkConf has a configuration parameter that our Spark driver application will pass to SparkContext.

In Spark 2 you can use SparkSession instead of SparkContext. In Spark 2.0 onwards, it is better to use SparkSession as it provides access to all the Spark functionalities that SparkContext provides. Also, it provides APIs to work with DataFrames and DataSets.

The following code comes in handy when you want to create a SparkSession;

If we write local [*] in the master, it will use all cores in our machine. If we said local [4] it will work with 4 cores. It can work in local / yarn / mesos and kubernetes mode.

getOrCreate is used to check whether a SparkSession exists. It creates a new one if not.

If we do not import SparkConf, we will have to assign complex metrics for beginners like “executor.memory” and “driver.memory” values. In such cases you can determine the configuration values as in the example below.
Finding Duplicate E-Mail Id using Snowflake and py-spark : Leet Code Problem : Category : Easy : Can you do it ?
By: Sanjit Khasnobis
Tags: Data Engineering, Sql, Snowflake, Pyspark, Duplicate

Finding Duplicate E-Mail Id using Snowflake and py-spark : Leet Code Problem : Category : Easy : Can you do it ? Sanjit Khasnobis 6 days ago·3 min read

Today we are going to try a very simple problem and try to solve it using both snowflake and py spark. Just to mention this is an common problem as described below -

/* Write a SQL query to find all duplicate emails in a table named Person.

+ — — + — — — — -+

| Id | Email |

+ — — + — — — — -+

| 1 | k@n.com |

| 2 | s@j.com |

| 3 | k@n.com |

+ — — + — — — — -+

For example, your query should return the following for the above table:

- — — — -+

| Email |

+ — — — — -+

| k@n.com |

+ — — — — -+

*/

We will first try to solve it using snowflake. First let us load the sample data to snowflake as below -

Data will look like as below -

Duplicate Email Id Data

Now below is the Snowflake Query to find the duplicate Email — Ids

Select mail from

employee_mails

group by mail

Having count(*) > 1

The data will look as below -

So except Samy and Ramy all are having duplicate email Ids

Now we will try the same problem with py-spark

Initiate the sparksession as below-

# importing sparksession

from pyspark.sql import SparkSession

# creating a sparksession object

# and providing appName

spark = SparkSession.builder.appName(“pyspark_findduplicate_function”).getOrCreate()

populate the data in a pyspark data frame as below -

The data will look as below when we print the data frame -

Except Samy and Ramy all Email Ids are duplicates

Now lets try to find out the count of email ids for each employee.

employeeemail_new = employeemail.groupby(“mail_id”).count()

employeeemail_new.show()

So now the new data frame will look like below -

Count of Email Ids

Now let us pull only email Ids which are having count more than 1 to fetch only duplicate Email Ids.

from pyspark.sql.types import IntegerType employeeemail_new.select(“mail_id”).where(employeeemail_new[“count”].cast(IntegerType()) > 1)

Duplicate Email Ids

So we have find the solution using both snowflake and py-spark.

Happy Coding! Happy Reading!!
Big Data Handling With PySpark
By: Syeda Marium Faheem
Tags: Big Data Analytics, Apache Spark, Pyspark Joins, Big Data Medium, Pyspark

Picture from towardsdatascience

Using python, you cannot only write the script, but also automate the boring stuff, and develop GUI and website. but most of its a super exciting feature you can find in ML and big data. Python is widely used in big data technology

Big data Engineers do functional programming writing scripts in a functional paradigm makes parallel code for processing

What is pySpark?

Apache Spark is an open-source computational engine use for big data processing .spark supports both batch and streaming processing. Spark is written in scala but python has its support python. In short, pyspark Is a python API. Spark supports python, R, and scala In Spark, we have several clusters. we can run some piece of code on one or multiple resources .when we are dealing with big data parallel processing makes life easy. In spark python, we just supposed to write functional code it will automatically distribute across the cluster

Spark’ Features

In this pyspark tutorial, we will cover pyspark big data from basic to advance

To follow this pyspark tutorial, we will cover everything from how to install pyspark to advanced big data concepts .we will cover important spark application examples you don’t need to install anything. To learn pySpark we will use google collapse python notebook or you can use pyspark Jupiter notebook

How to install Spark:

pip install pyspark

Upload File on Google Collapse Using Python API
Keycloak integration: Part 1: Nginx and Keycloak(proxy) deployment
By: Apoorvaditya Varma
Tags: Docker, Nginx Proxy, Keycloak, Nginx

Integrating the Keycloak as a reverse-proxy server in our webserver of Nginx can be a useful setup. There are other blogs on this but I faced lots of challenges in setting the Keycloak in the desired way, so writing this blog series where I go through setting Keycloak with different types of clients in Realm this is First Part of the series and this to deploying Keycloak with Nginx Webserver.

In this blog we will be setting the Keycloak and Nginx in the following way:

Fig 1. HTTPS and HTTP request flows in setup.

Keycloak server deployment and configuration with PostgreSQL

Note:

1. Create the respective docker network and volume for the services.

2. Keycloak service:

2.1. You can also use schema base table creation with PostgreSQL (you also need to update your database also "CREATE SCHEMA kc_model;") and update the environment variable like-

DB_SCHEMA: kc_model

2.2. You can further define your frontend-URL for admin console or also pass using the environment variable like-

2.3 JDBC Params can also be added this will be appended to Keycloak’s DB-connection URL below is the log output from Keycloak:

"connection-url" => expression "jdbc:postgresql://${env.DB_ADDR:postgres}/${env.DB_DATABASE:keycloak}${env.JDBC_PARAMS:}"

Use below docker-compose file for managing Keycloak, PostgreSQL, and Nginx

Also, add the webserver_nginx.conf file, update the path in docker-compose if required assuming it to be in same directory with compose file.

Nginx Configuration for Keycloak Upstream

Further, you need to add the SSL certificate data to the /config/cert location.

Further to remove the containers, volumes and network follow

docker-compose -f docker-compose-kc-psql-nginx.yml down

docker volume rm psql_storage

docker network rm my-net

# and the downloaded file

Conclusion:

By the above setup now we can reverse proxy our Keycloak server and this can be accessed by domain.com/auth .

PS: More parts will be added to this Keycloak Integration blog series.

Happy Coding. 👨‍💻
OpenSSL CSR with Alternative Names
Tags: Openssl, Https, Nginx, Ssl Certificate

What is a CSR (Certificate Signing Request)?

A CSR or Certificate Signing Request is a block of encoded text that is given to a Certificate Authority when applying for an SSL Certificate. It is usually generated on the server where the certificate will be installed and contains information that will be included in the certificate such as the organization name, common name (domain name), locality, and country. It also contains the public key that will be included in the certificate. A private key is usually created at the same time that you create the CSR, making a key pair.

What is contained in a CSR?

What does a CSR look like?

Most CSRs are created in the Base-64 encoded PEM format. A PEM format CSR can be opened in a text editor and looks like the following example:

-----BEGIN CERTIFICATE REQUEST-----

MIIByjCCATMCAQAwgYkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh

MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMR8w

HQYDVQQLExZJbmZvcm1hdGlvbiBUZWNobm9sb2d5MRcwFQYDVQQDEw53d3cuZ29v

Z2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEApZtYJCHJ4VpVXHfV

IlstQTlO4qC03hjX+ZkPyvdYd1Q4+qbAeTwXmCUKYHThVRd5aXSqlPzyIBwieMZr

WFlRQddZ1IzXAlVRDWwAo60KecqeAXnnUK+5fXoTI/UgWshre8tJ+x/TMHaQKR/J

cIWPhqaQhsJuzZbvAdGA80BLxdMCAwEAAaAAMA0GCSqGSIb3DQEBBQUAA4GBAIhl

4PvFq+e7ipARgI5ZM+GZx6mpCz44DTo0JkwfRDf+BtrsaC0q68eTf2XhYOsq4fkH

Q0uA0aVog3f5iJxCa3Hp5gxbJQ6zV6kJ0TEsuaaOhEko9sdpCoPOnRBm2i/XRD2D

6iNh8f8z0ShGsFqjDgFHyF3o+lUyj+UC6H1QW7bn

-----END CERTIFICATE REQUEST-----

How do generate a CSR and private key?

copy default settings for further editing

cp /etc/ssl/openssl.cnf /etc/nginx/ssl/bookstyle.cnf

open bookstyle.cnf find [ req ] uncomment req_extensions = v3_req

find uncomment find the block [ v3_req ] and at the end add

subjectAltName = @alt_names

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = @alt_names

add new block [ alt_names ] where you need to specify the domains and IPs as alternative names

[ alt_names ]

DNS.1 = *.bookstyle.com

DNS.2 = bookstyle

IP = 192.168.15.247

create a private key and a certificate signing request by using config and send bookstyle.csr to your CA

openssl genrsa -out bookstyle.key 2048 openssl req -new -key bookstyle.key -out bookstyle.csr -config bookstyle.cnf

place the received bookstyle.cer file from your CA in needed folder, specify path for this certificate and private key in

vi /etc/nginx/ssl_nginx_tlsall_params

if needed configure vi /etc/nginx/nginx.conf

restart the nginx systemctl restart nginx

By the end you will enjoy a green security lock view in your browser in accessing via https the links:

https://mail.bookstyle.com/

https://bookstyle/

https://192.168.15.247/

Reference
Automate your Docker Container Deployment with Terraform [For Mac OS]
By: Tharindu Abeysekara
Tags: Macos, Infrastructure Automation, Docker, Nginx, Terraform

First, we are going to Install Terraform and Docker on your local pc.

Open your terminal and install HashiCorp on your MacBook.

$ brew tap hashicorp/tap

Then install Terraform using the below command.

$ brew install hashicorp/tap/terraform

To update the terraform to the latest version, first, you need to update Homebrew.

$ brew update

Then you can Update Terraform with the below command.

$ brew upgrade hashicorp/tap/terraform

Now you can verify your installation using the below command.

$ terraform -help

If your installation is successful, you will see a similar result as the below screenshot.

result of “terraform -help” command

To Enable tab completion for auto-complete the commands. First, check for the config file on your shell.

$ touch ~/.bashrc

Then install the terraform autocomplete package using the below command.

$ terraform -install-autocomplete

Before we continue, you need to download and install the Docker Desktop for Mac from the below URL. [ System Requirements]

Download: Docker Desktop for Mac

After you install Docker on your computer run the below command to start Docker Desktop.

$ open -a Docker

Now you have successfully completed the Terraform and Docker installation.
FALSE: The Chinese Government has not threatened to deport Kenyans infected with COVID-19
Tags: Coronavirus, China, Misinformation, Fact Checking, Kenya

FALSE: The Chinese Government has not threatened to deport Kenyans infected with COVID-19

An article published by Nairobi Times claiming that the Chinese Government has threatened to deport Kenyans with novel coronavirus in retaliation to an order to deport Chinese workers in Kenya who lacked proper documentation is FALSE.

According to the article, China has threatened to deport Kenyans in Wuhan infected with COVID-19 in retaliation to Interior Cabinet Secretary Fred Matiang’i’s order to deport four Chinese workers who were found to be working in Kenya illegally. One of the workers was caught on camera caning a Kenyan waiter at a restaurant in Nairobi.

The article claims that the Chinese Supreme Court issued a note to CS Matiang’i asking him to find other legal avenues to deal with the four Chinese rather than deporting them. The article also does not indicate key details, such as when the note from the Supreme People’s Court was sent, and who specifically wrote it.

While the article refers to the ‘Chinese Supreme Court’, it does not have the specifics of who at the court or in the Chinese government had given this order. Additionally, the court is officially known as the Supreme People’s Court, and does not contain any such communication on its official website.

There are no reported cases of Kenyans infected with novel coronavirus since the outbreak started according to the World Health Organization. There have been a number of suspected cases, but none has tested positive.

The deportation order referenced in the article was issued on 13 February by Interior Cabinet Secretary Fred Matiang’i, but was suspended by High Court judge Luka Kimaru on 17 February.

Additionally, the claim that the Government of China had threatened to deport Kenyans in Wuhan has not been published in any mainstream news outlets, and the government has not mentioned it in any of their briefings.

In a press briefing on 20 February, Government Spokesperson Cyrus Oguna announced that the Kenyan government has released Ksh.1.3 million for students who are trapped in Wuhan, China following the COVID-19 outbreak, adding to a Ksh. 500,000 donation by the Chinese Government to help the students get supplies.

There was no mention of deporting Kenyans in Wuhan, which would have been communicated during the press briefing given its significance.

Elseba Chepleting, a Kenyan lawyer, told PesaCheck that any communication from the Chinese Supreme People’s Court to the Government of Kenya would be through the ambassador to a court of equal jurisdiction, in this case the Supreme Court of Kenya, and not directly from the Court to the cabinet secretary as indicated in the article.

There is no such communication on the website and the Twitter account of the Embassy of China in Kenya.

PesaCheck has looked into the claim that the Chinese Government has threatened to deport Kenyans infected with novel coronavirus and finds it to be FALSE.
WARNING FOR IMMEDIATE CORRECTION — WSJ Editorial Falsehood Re: Susceptibility of COVID-Recovered To Re-infection
By: Hooman Noorchashm
Tags: Wall Street Journal, Misinformation, Covid 19 Vaccine, Coercion, Natural Immunity

Messrs., Gigot, Murrary, Henninger and colleagues,

Today, I was shocked to read what I know, with 100% certainty as an immunologist and physician, is a lie in the WSJ’s Editorial Board OpEd on COVID-19:

https://www.wsj.com/articles/covid-delta-variant-market-dow-industrial-700-points-11626731103?mod=hp_opin_pos_1

Specifically, your colleagues are stating the following falsehood:

“Previously infected individuals appear to be more susceptible to re-infection by the Delta variant, which could explain some of the rising cases.”

This WSJ editorial statement is a dangerous lie, with NO basis in either current scientific epidemiological data or the fundamentals of immunology.

To the contrary, the the bulk of studies on the topic of “re-infection” in the COVID-recovered, are demonstrating that individuals who are COVID-recovered and have thus acquired Antibody and T-cell immunity to SARS-CoV-2, are equally if not better protected from subsequent infection as compared to the vaccinated. Though I am cognizant this fact ought NOT be abused as an argument for seeking natural infection as a pathway to immunity, it IS the argument for not allowing coercion of COVID-recovered Americans into undergoing what is an unnecessary and potentially dangerous treatment to them.

I am left wondering why the editorial board of a respected media outlet like the WSJ would allow itself to spread such a lie with no stringent verification:

1) Are you simply careless journalists now? or

2) Are you now active participants in the US Government’s publicity machine attempting to impose a “one-size-fits-all” vaccine policy on all Americans, irrespective of medical necessity, the fundamentals of immunological science and medical ethics?

Irrespective of what the reason is for the lie you have now published in today’s editorial article, it is an absolute journalistic duty for you to either correct or retract this false statement immediately— given that you have been alerted to it.

For the WSJ to publish a critical false statement that opens the door to misleading the public and galvanizing generalist politicians and regulators into comfortably accepting unnecessary, and potentially dangerous coerced vaccination of the already immune subset of Americans, is a florid and dangerous dereliction of journalistic duty — but, of course, this seems to the norm for the mainstream media these days.

Please immediately retract your editorial board’s false statement and correct it to reflect the truth of what is known — and what the science of immunology actually would predict. Your editorial piece, as it stands, promotes a very serious lie to the American people.

Sincerely,

Hooman Noorchashm MD, PhD

noorchashm@gmail.com
FALSE HEADLINE: Kenya has not reported a case of Ebola in Kericho
Tags: Ebola, Misinformation, Fact Checking, Kenya, Verification

FALSE HEADLINE: Kenya has not reported a case of Ebola in Kericho

The headline that Kenya has found a case of Ebola is contrary to the statement issued by Kericho County PesaCheck Follow Jun 17, 2019 · 4 min read

A story published by Somali Top News and shared on Facebook claiming in its headline that Kenya has found a case of Ebola in Kericho has a FALSE HEADLINE.

On 16 June, the Office of the Governor in Kericho issued a statement informing the public of a patient admitted at the Kericho County Referral Hospital exhibiting symptoms of Ebola. The statement further read that the 36 year old patient had been quarantined and her blood samples sent to the Kenya Medical Research Institute (KEMRI) for analysis. However, the statement emphasized that there was no confirmed case of Ebola in the county, contrary to the article published by Somali Top News.

In January 2019, PesaCheck debunked the claim that two cases of Ebola had been reported in Nairobi.

According to the American Red Cross, over 1,400 have died from Ebola in the Democratic Republic of Congo (DRC) since August 2018.

Uganda recorded the first case of the virus in early June 2019, putting Kenya and other neighbouring countries on high alert.

The Ministry of Health has since confirmed that that the female patient quarantined at the Kericho County Referral Hospital has tested negative for the Ebola Virus.

PesaCheck has looked into the article titled: Kenya found Ebola case in Kericho” and finds that it has a FALSE HEADLINE.