fabiochiu/medium-articles huggingface dataset:
Apache Hive Hooks and Metastore Listeners: A tale of your metadata
By: Marcelo Costa
Tags: Big Data, Metadata, Data, Data Warehouse, Hive

The Need for Metadata Management

According to Gartner “By 2020, most data and analytics use cases will require connecting to distributed data sources, leading enterprises to double their investments in metadata management.”

After reading that statement, you are probably wondering: What does metadata really stand for? And why does it need Management?

I can tell you those were the first questions that came to my mind a few months ago…

Metadata

We can categorize metadata in:

Technical Metadata : details about an asset, such as its name, type, the creator's name, the size of the object, or when it was last updated.

: details about an asset, such as its name, type, the creator's name, the size of the object, or when it was last updated. Business Metadata : offers additional business context about an asset, for example, if it contains PII (Personally Identifiable Information), the date it should be deleted, and many others.

: offers additional business context about an asset, for example, if it contains PII (Personally Identifiable Information), the date it should be deleted, and many others. Operational Metadata: usually information on how your assets are being used, like query logs, data sampling and data profiling.

You can also think about physical and logical information about the asset, let me clarify that by looking at Hive and the Hadoop Ecosystem…

Your data can be physically stored in a hdfs location, and have many different logical tables pointing to it on Hive.

Photo by Amy Chen on Unsplash

That’s a lot to process, but it will become clearer later on…

Metadata Management

A Metadata Management solution helps answering questions related to your data assets such as:

Are my data assets secure?

Am I in compliance with all those new data protection regulations? Such as CCPA, GDPR, HIPAA

Who has visibility and who can make changes to those assets?

The nature of management usually starts with a wiki, spreadsheet, and various documentations, but when we start to enter the big data world, they can easily get out of sync and we become unable to answer those questions.

Do you remember the $5 billion fine regarding Facebook user's data privacy? You don’t want to be put in a place unable to answer those questions.

We need to start automating our metadata management!
Asking for Free Writing Advice is The ‘Send Nudes’ of My Profession
By: Sean Kernan
Tags: Humor, Life, Business, Life Lessons, Writing

Attack of the LinkedIn Inbox

“Can you look at my writing and provide some feedback?”

“Can I have some guidance on your content strategies?”

“Hey, would you review my draft?”

Welcome to my inbox. This is a good chunk of the messages.

There’s a noticeable brazenness with the writing profession, a willingness to hawk professionals for free stuff. Perhaps it's because ‘anyone can write sentences’ and everyone has written essays in school, occasionally scoring a good grade.

Perhaps they assume writing has this airy, easily tangible quality, like picking up a pen someone just dropped.

Other times, their inquiry is more calculated. That’s when my claws come out. They’ll say, ‘I’d love to pick your brain’ or ‘meet for coffee’ when they are just looking for free consulting.

“I’ll buy my own coffee and charge my rate. Cool?”

Back in my agency days, clients occasionally kicked and screamed about billings. There was the old, “Why am I paying this much for something your graphic designer can do in a couple of hours?”

The blunter version of our response, “You’re paying for the years of experience that enabled them to do it in a couple of hours. And the fact that this designer is very talented.”

This disconnect, the lack of value for artistic merit isn’t a new phenomenon. It’s also why most company blogs are little more than the walking dead. Their marketing team bottom-feeds and they get what they pay for.

Anyone can type words. And anyone can wield a paintbrush.
No, political interests have not warped the vaccine approval process.
By: Max Ufberg
Tags: Covid 19, Health, Vaccines

No, political interests have not warped the vaccine approval process. Over at Elemental, journalist Tara Haelle spoke with medical experts about the coming Covid vaccine and why we should feel secure that a vaccine’s distribution is a reflection of its efficacy — not a politician’s deep pockets.

“Safeguards are in place to really limit the ability of any one scientist in being unethical,” says Michele Andrasik, the director of social and behavioral sciences and community engagement at the Fred Hutchinson Cancer Research Center’s Covid-19 Prevention Network. Among those safeguards: FDA and CDC advisory panels, and mandatory trial pauses for vaccines that appear to have adverse effects. (Both AstraZeneca and Johnson & Johnson were forced to halt their trials in September after subjects reported adverse effects.)
The Naughty Reason Nails Were Vanishing from the HMS Dolphin
By: Sean Kernan
Tags: Health, Travel, Life, Life Lessons, History

The disappearance of iron nails

Captain Wallis frequently went ashore with his cartographers to get proper documentation of the island, as was part of his exploratory orders.

He also mingled with the locals and communicated with Tahitian leaders. Outside of the initial fighting, the people were surprisingly kind and generous. Many meals were shared together.

But several weeks in, he noticed iron supplies disappearing from the ship. Notably, most of his spare iron nails. They were critical for any mid-transit repairs. Even worse, his first mate pointed out several nails that’d been pried out of the main hull itself.

Wallis soon discovered the reason and was infuriated. Upon their arrival, his sailors were instantly been smitten by the beautiful Tahitian women. This in and of itself wasn’t a problem. But the women offered the sailors sex in exchange for iron. It was a coveted metal on the island.

Author via Pinterest

Additionally, in the 18th century Tahitian culture, before Christian missionaries pushed their morality on the islanders, women were free and empowered to make such offers with their bodies. So much so, that even Tahitian men were puzzled that visitors didn’t have such an open society (or women on their boat).

The sailors, known for excess, burned through all the ships available iron and even tested the very nails that held the ship together.

Wallis was aghast at the limitless bounds of his men’s lust. Like many captains of his stature, he was an educated British man, cultured, reserved, and placing value on restraint. However, he knew he too bore blame for being so lenient with his men.

Wallis brought the men in and threatened lethal punishment on the next to touch another morsel of the ship’s iron.

He resolved they’d leave by week’s end and did so, returning home in one piece.
The Sacred Void in the Election
By: Nick Keehler
Tags: Philosophy, Religion, Election 2020, Society, Politics

The Sacred Void in the Election

Where do we go once the dust settles down?

Photo by Jon Sailer on Unsplash

As I write this, the 2020 presidential election results are still trickling in. I walk through the living room and see my family watching intently as it unfolds.

“Why are you watching?” I inquire. “You can just wait a day or two and find out who won.”

“It’s the drama,” they respond. “The anticipation. It’s like watching sports, except the winner has lasting implications on our lives.”

I’ve never been one to watch sports, so I nod with acceptance, grab the bag of chips I came for, and make my way back upstairs. Throughout all the buildup to the election, I was constantly drawn back to a lecture I read during college. Religion and Secular Culture by Paul Tillich.

Though it has been years since I read it, it still stews in the back of my mind. Though it was delivered in 1946, his words still ring as true as they ever have.
Dolly Parton’s “Together You and I” Video
By: Edith Zimmerman
Tags: Music, Dolly Parton, Music Videos

We first heard this searing ray of sunshine back in May (or in 1974, when she sang it with Porter Wagoner), and it now has a touchingly cheesy video, in which Dolly Parton — Glinda the Good Witch — unites the entire planet. Dolly Parton for General Queen. (Her Better Day album is out now.)

[Via]
5 Fitness Tips to Improve the Quality of Your Nutrition
By: Jeremy Colon
Tags: Health, Nutrition, Metabolism, Fitness Tips, Weight Loss

Photo Credit: Jeremy Colon

1. Plan and Prepare

The fitness champs ensure their balanced meals are pre-prepped to avoid getting hungry and reaching for an unhealthy meal. Do a big cook-up once or twice a week and carry healthy meals and snacks on your person to help avoid temptation.

2. Have Carbs Earlier in the Day

Have most of your high-energy carbs earlier in the day. Think your first three meals. Front loading your carbs provides energy for the day ahead and can offer clues as to how well your body responds to types and amounts of carbs.

3. Eat Regularly

Think every two to three hours. Eating regularly helps to manage your hunger hormones ghrelin and leptin, dousing the temptation to reach for that sugary chocolate bar.

4. Don’t Be Afraid of Hunger

Don’t be afraid of hunger; be afraid of undereating. Competitors are hungry all the time because they’re training hard and their metabolism is in overdrive. Hunger is a good sign — if you can eat in response and don’t feel deprived. Nourish your body and eat the calories that your body needs to function effectively.

5. Supplement Smart

Don’t go spending your hard-earned cash on a mountain of sups you don’t necessarily need. The pros supplement for their specific goals, current body composition, and deficiencies. Start with a high-quality multivitamin and fish oil tablet, and then work with your trainer to fill the gaps.

If reading this offered you value, please recommend it and share it with others! Also please connect with me on my website, Facebook page, and Instagram!
An Introduction to Data Cleaning: Using Regular Expressions to Clean your Data
By: Johnathan Padilla
Tags: Air Quality, Data Cleaning, Python, Data, Regex

Johnathan Padilla, Thomas Kidd, and Hadrien Picq are fellows in the TechSoup and ParsonsTKO Summer 2020 Data Strategy Mentorship Program, a select group of upcoming data analysts and scientists to learn more about what the social impact and non-profit analytics sectors look like from industry professionals.

Regex is a Superpower

Unlike powers inherited by radioactive spiders, regex can be used as irresponsibly as you want. There are limits to this power but when used skillfully those limits can be largely ignored. I’ll show you a quick example of regex’s power. Say you are given a string like the following:

Words = 'the quick brownTfox runs super! fast: '

The sentence is all garbled and has characters that we do not want to keep. We can correct and recreate this sentence using regex. First, you have to split the string on the characters of your choice.

split = re.split('T|!|:',Words)

split

I chose to split on the ‘T’, ‘!’ and the ‘:’ which yields a list of only the desired characters

['the quick brown', 'fox runs super', ' fast', ' ']

From here I would want to recombine the strings to one string. I can do this by adding each portion of the list to each other like so

corrected_words = split[0] + split[3] + split[1] + split[2]

corrected_words

Be sure to note that I placed the space character in between ‘brown’ and ‘fox’ as no space existed there in the first place. Printing the new string should look like this:

'the quick brown fox runs super fast'

Using the regex method is a faster way of cleaning text data that is simpler and quicker to use compared to manually splitting the strings. Now that we have established a basis for using regex I can dive into the data being used for the project.
The Holiday Olympic Torch
By: Emma Holiday
Tags: Society, Creative Non Fiction, Humanity, Love, LGBTQ

The Holiday Olympic Torch

Passing the burning torch of hope

When the world holds the Olympics thousands of people from around the globe carry the Olympic torch, handing it off, one by one, as a symbolic gesture of peace and solidarity. I see the holidays in the same way. I know for many it is a time of stress, for some a time of personal pain, for others a reminder that their life has failed to provide them with the means to celebrate while there are others that lack the basic needs of life.

Unfortunately, it will always be that way. Life does carry with it a certain cruelty. All my life I have done things to make a difference but people continue to suffer. I have come to realize that I am not Mother Theresa nor a doctor with the Doctors Without Borders nor a UN relief worker nor the countless others that are willing to sacrifice all for the good of others. I have thought about it many times but it is not something I am capable of.

I mentally apologize and accept my human limitations.

That doesn’t mean I don’t contribute to improve the quality of life for others. I do so every day in small things and occasionally in bigger ways. It allows my heart to express love.

Every time I do something, I hope that I am passing the burning torch of hope and love onto others and that they will, in turn, pass it on. I have always subscribed to the idea that the best acts of kindness and charity are the ones done privately and without fanfare.

Those are the best ones to witness.

At the same time, each act is the torch of humanity, multiplied with each act as the torch is handed off to others, directly or indirectly, to create a chain of kindness, shared humanity and love around the world. The torch seems to burn brighter during the holidays as we all open up our hearts. This pandemic and the global political strife we are witnessing, unfortunately, has added additional darkness to our world.

We need an even brighter torch this year.

Watch for your opportunity to take the torch and carry it for just a brief moment and then willingly pass it on. The more times we carry it, the more times we share it, the brighter the world will be.

May the light and the warmth of the holidays carry you well into the New Years and beyond.

Emma Holiday

This story is a response to Prism & Pen’s writing prompt “Blaze Against the Dying of the Light”

Other stories so far —
What Makes Social Media, Social Media?
By: Aldric Chen
Tags: Communication, People, Personal Development, Technology, Social Media

What Makes Social Media, Social Media?

In short — People and communication.

Photo by ROBIN WORRALL on Unsplash

Social media is 90% social, 10% media. These technology platforms allow for infinite power and reach. Our work can reach millions within a fraction of a second after hitting the “Post” button. More people can access our work concurrently, and more than ever before.

As much as I see the technological prowess which connects millions at any particular point in time, many see the good and the dark side of social media.

Let me stick to the technological layer for a bit before I touch on the social element.

These web and mobile applications have a wider outreach compared to any physical platforms we knew. Television, billboards, and posters have one thing in common.

They cannot move.

Smart devices can move. Through these physical proxies we carry with us like our wallets, the social media platforms follow us wherever we go.

Getting our attention is one vibration away. Visual assaults or on-air commercials could only be envious of their younger but more powerful step siblings.

Now, leaving the technology layer aside. The social element makes up the social media we know today.

If we could examine all those platforms we are involved in, we can find a common thread. They are designed to facilitate and exaggerate communication.

Like a trendy drama, the main cast is ridiculously handsome, rich beyond means, taller than an NBA player, gyms, have eyes like a hawk, and can run like a cheetah.

When we engage in social media, we express our love equivalent to the magnitude of an earthquake. We comment to approval, like, share, and funneling in others based on tagging on that same post.

That’s social media.

The dark side of social media is manifested in communication as well.

I think an influencer can bury those they dislike. This is nothing new. In a corporate setting, those on top could end the career of a certain someone at the operational level.

In that sense, humans are humans. We carry the same behavior wherever we go.

What social media actually does is to extend the reach of our behavioral and communication patterns beyond our physical boundaries.

We can send our love to a child in Siberia instantaneously. We can also continue to destroy that person we hate no matter where they escape to. Zambia or Columbia doesn’t matter. The throttling continues so long as social media accounts are alive.

This is something that we have to adapt to in the 21st century. Social media will continue to evolve, and we have to learn to deal with it.

That is the same with dealing with difficult people, annoying friends, and seemingly scheming adversaries.

In that sense, social media is as normal as it is novel.

This is the social media we know today.

Social Media Is A Reflection Of Social Element,

Aldric
Important life lesson from a Halo 4 developer
Tags: Leadership, Perspective, Time, Goals, Creativity

You don’t have to be a video game fan to appreciate this. No…seriously.

Halo is one of the biggest video game series in history. Master Chief, the main character, is up there with Mario, Pac Man, Solid Snake, and Zelda. Halo 4, the newest chapter in the saga is currently in development and is for sure a highly anticipated title.

I read this morning that Ryan Payton, one of the creative directors for Halo 4, is stepping down from his position. This is obviously a big blow for the game’s development. It also begs the question as to why he’s doing it.

The whole article about his decision can be read HERE, but this is the part that really struck me:

“For somebody who loves this industry as much as I do and know how lucky I’ve been, I never thought I’d get to a point where I was so drained. That was when I knew I had to do something else,” said Payton. “I think time is the most valuable thing we have, and I’ve decided that I’m not going to waste one more day working on something that doesn’t speak to my values.”

Like I said, you don’t have to be a gamer for that bolded line to hit home.

Can you articulate your core values?

Does how you spend your time reflect your core values?

What are you wasting time on?

Answering those questions will take courage. A ton of it.

Ryan’s statement reminded me of a song by Chris Rice, “Life means so much.”

If you have time, it’s worth a listen:
We don’t know what we eat… Only Blockchain can bring the solution.
Tags: Food, Health, Health Foods, Blockchain, Covid 19

Broken supply chains and rising food prices

All countries are currently facing with food price surges and shortages due to broken supply chains. In this turbulent context, it is often impossible to make sure that the remaining agricultural products available on our stalls are actually safe to eat, free from poisonous crop protection residues.

Then, does it really make a difference consuming products while being able to trust crop protection, harvest, storage and shipping methods with our lives ?

The obvious answer is YES. In fact, unhealthy food production processes are responsible for an increasingly important number of casualties, projected to be over 5 million deaths a year by 2050.

The only solution, Blockchain

Then, how to trustfully isolate the good from the bad products? How to make sure all agricultural products, including animals, are given the care they ought to? Only one answer: Blockchain.

Imagine an immutable, decentralized and always accessible database. A real travelogue containing all production details, accessible as easily as scanning a QR code on the final product.

At Fieldcoin, we are able to extract such critical informations by providing 4.0 Agribusiness technologies to our farmers. This way, they benefit from an extensive use of these state-of-the-art technologies to greatly improve and fully monitor their yield, while being 100% transparent with the consumer.
This Christmas, It Is Well with My Soul
By: James Finn
Tags: Family, Joy, Grief, Christmas, Music

Joy amidst sorrow —

In 10 Things You Probably Don’t Know about Me, I wrote that people might be surprised to find out that I love traditional Christian hymns like Amazing Grace and It is Well with my Soul.

Have you ever heard It Is Well With My Soul by Phillip Bliss and Horatio Spafford?

The music is deeply sad in tone and feeling. I’m not musically educated enough to explain why. I just know it touches something deep inside me, even without knowing the story and hearing the lyrics.

But those lyrics! Here’s the first verse and the chorus:

When peace like a river, attendeth my way,

When sorrows like sea billows roll;

Whatever my lot, Thou hast taught me to say

It is well, it is well, with my soul. Refrain:

It is well, (it is well),

With my soul, (with my soul)

It is well, it is well, with my soul

And here’s a sung version I find incredibly moving —

The story?

Horatio Spafford, a middle-aged American businessman, penned those words one dark, freezing night in 1873 on a ship in the middle of the Atlantic — as he steamed past the spot where only weeks before, his four daughters had perished in the wreck of the passenger ship Ville du Havre.

He’d received the news in a two-word telegram from his wife, “Saved alone.”

Spafford penned this plaintive yet defiant poem as his soul cried out in grief and despair. He would grieve, but he would not give in. He would absorb the blow and he would be well.

Bliss composed perfect music for the piece. It makes my own soul vibrate in electric sympathy.

I’m hanging on to that this holiday season, as best as I can.

It is well with my soul.

Perhaps if I repeat that phrase often enough, and listen to the hymn often enough, I’ll find truth in it.
The Rise of Hijabi Bloggers
By: Islamic History Notes
Tags: Modest Fashion, Hijab, Family, Muslim Fashion, Creativity

A phenomena of halal celebrities is beginning to arise and is taking the Muslim fashion scene to new levels. Women that we had never heard of are now common names among Muslimahs. These women have mastered social media, display unique fashion sense and some of them even maintain a side business.

Admirers of these various blogs and Muslim Fashionistas have said that they finally feel they can be fashion forward while still maintaining their religious identity.

But critics of these personalities say that the Muslim bloggers are in fact slaves to the fashion industry and are promoting the objectification and sexualization of hijab and modest fashion

Let me take you back ten years when I first started wearing the hijab. Triangular gray, black and white scarfs pinned at the neck with the two ends tied in the back. Anyone remember that one? That style and design was neither appealing nor attractive in any manner. It really was a struggle to wear it in high school when all the other girls looked so put together.

At that point had I seen some of the Amenakin (Pearl Daisy) hijab tutorials in which she beautifully incorporates the tikka, (decorative jewelry that hangs in the middle of the forehead) I wouldn’t be awkwardly stumbling around Pakistani weddings wearing a grandma style dupatta.

I know of many girls who take off their hijabs either before or after marriage, and in some cases may even feel hijab is the reason why they aren’t getting proposals. As Muslim women we have a fine balancing act, between modesty and beauty.

Having access to muslim bloggers who can offer creative ways to style modest clothes and hijabs can be an asset. They are not self-proclaimed experts but normal Muslim women who usually have been approached by their fans to tell them about their skincare regime, weight loss tips and how to get that smokey eye right.

I may not agree with everything that Muslim fashion bloggers promote but I do think there are many ideas and tips which are creative and inspiring. The hijab and the act of dressing modestly is a personal journey for each women and it is something which each of us can improve on.
Creating a dynamic DAG using Apache Airflow
By: Antony Henao
Tags: Data Science, Airflow, Data Engineering

Today we want to share with you one problem we solved by using Apache Airflow. We have a project comprising more than 40 apps. Every day we have to load data from on-premise databases to the cloud—particularly, to AWS S3. The process is performed in batch and executed every day. Due to particular reasons, the data has to be loaded to S3 without any further pre-processing. Tables to be loaded to AWS S3 may vary regularly, so it should be easier to add such tables as input sources — since we don’t want to make a DAG for every table we have in the databases.

We decide to solve the problem described above by automatically generating an Airflow DAG. Particularly, we design the solution so the DAG workflow can be generated from a simple YAML, i.e., given a YAML file containing table names to be loaded to AWS S3, Airflow should automatically generate DAG tasks for loading such data. So, in this post, we want to take you through our solution. We hope you may find it useful!

YAML file

Tables to be loaded to S3 should be specified in the YAML file. Furthermore, since it’s a batch process, we need to include a date-type field for filtering such tables based on the execution date. So, we perform incremental loads, i.e, only data from the day before is loaded into S3.

Dynamic Task Generation

Once the YAML file structure is defined, we can build the logic for our dynamic DAG! So, the first thing to do is defining two tasks using dummy operators, i.e., the start and the end task. Such tasks are the ones in which we are going to build upon our DAG by dynamically creating tasks between them — at this point this may be a little confusing, but once you see the graph everything is going to be clear.

Then, next is step is defining a Python function allowing for creating DAG tasks. Particularly, we create such tasks by using PythonOperators . The function should receive as arguments the task id; a python function to be executed, i.e., the python_callable for the Python operator; and a set of args to be used during the execution.

We include as an argument the task id. So, we can exchange data among tasks generated in dynamic way, e.g., via XCOM.

Setting the DAG workflow

The final step is putting together the workflow for the DAG, which is really easy if you got to this point.

This is how our DAG looks like after putting the code together

DAG automatically generated by using a YAML file

Conclusions

In this post, we present a way for creating dynamic tasks inside an Airflow DAG. We let out this article the definition of the functions getSQLData and upload_to_s3_task, since we consider they are out of the scope of this article.

You can find the final code in the next snippet

Hope you find it useful.

Thanks for reading until the end! :)
My Journey From Open Source Noob to Google Summer of Code 2020
By: Shubham Kumar
Tags: Open Source, Gsoc, Google, Mozilla, Programming

My Journey From Open Source Noob to Google Summer of Code 2020

How to get started in open source

Photo by Alex Holyoake on Unsplash

What it means to contribute

If you’re a new open source contributor, the process can be intimidating. How do you find the right project? What if you don’t know how to code? What if something goes wrong?

Don’t worry! There are all sorts of ways to get involved with an open-source project — you don’t need to know everything just to get started.

For anything more than a typo fix, contributing to open source is like walking up to a group of strangers at a party. If you start talking about llamas, while they were deep in a discussion about goldfish, they’ll probably look at you a little strangely.

Before jumping in blindly with your own suggestions, start by learning how to read the room. Doing so increases the chances that your ideas will be noticed and heard.
Visualising 2018 tree cover loss with Global Forest Watch.
By: Camellia Williams
Tags: Forest, Environment, Data Visualization, Deforestation

Global Forest Watch allows anyone, including you, to take a closer look at what’s happening in places reserved for indigenous peoples. A fast, smooth zoom enables and encourages deeper exploration of what’s occurring. No one can hide from satellites in the sky and with weekly GLAD alerts, any change in forest cover can be detected and reported in almost real-time. This is the power of data visualisation: it provides compelling, irrefutable evidence that allows us to speak up for the people and places that are not being heard.

Good news for Indonesia.

But it’s not all bad news. Indonesia saw a 63% reduction in primary forest loss from its peak in 2016 when new legal protections of peatlands were introduced. More recent government policies—which include a moratorium on issuing new licenses to use land designated as primary forest and peatland—have also helped slow deforestation.

The clearing of rainforest to make way for oil palm plantations has attracted much media attention, and with a recently added ‘Global Plantations’ data layer, you can now use Global Forest Watch to explore where they are located. (Note: only data for 2015 is currently available). Although the moratorium is likely to stop any further conversion of pristine primary forest, GLAD alerts will help researchers, land managers, and interested citizens alike keep watch over Indonesia’s forests.
Dynamic resource generation of resources using CloudFormation Macros.
By: Pablo Perez
Tags: Macros, AWS, Dynamic, Cloudformation

One of the things you can do with this new feature is generate and deploy a number of resources based in a parameter value or the resulting value of the describing the number of AZs in order to create a subnet for each AZ for example.

In this example I’m defining a dynamic number of IAM users (depending on the int I provide in the parameter) that will have the same custom EC2 policy for EC2 resources tagged with Owner:devteam. The user will be prompted to reset their passwords on next sign-in. Also an EC2 instance will be created for each user simulating a classroom.

How to work with Macros:

In order to use this feature you need two things:

-One stack containing a AWS::CloudFormation::Macro resource along a “AWS::Lambda::Function”.

-The stack that will use the Macro which will process a section or the whole template by using on it the Fn::Transform function or a Transform section to transforming the whole template.

Biggest Caveat:

- Your Function will receive the whole template minus transform sections.

Take care of how you send the Macro Response in your lambda function.

You need to send the proper “Fragment”, this is the the processed template CFN receive from the lambda function.

def lambda_handler(event, context): FinalFragment= event[“fragment”] …loops adding new objects to the template.

…. …. … … …. response[“requestId”] = event[“requestId”]

response[“status”] = “success”

response[“fragment”] = FinalFragment print (FinalFragment[“Resources”]) return response

In the sample attached I’m using python and just looping to create the number of resources I need in the final template. However you can go further with your function and implement more complex logics using the SDK to do API describes etc…

You can keep several Macros in your account doing different types of processing and using them whenever you need unlimited power to process your template suiting to a huge amount of use cases.

Outcome:
Baby-Sitters Club Books as Classic Novels
By: Jack Shepherd
Tags: Literature, Books, Humor, Art, Television

Baby-Sitters Club Books as Classic Novels

The appropriate way to view Ann M. Martin’s contributions to literature

Now that a new generation is experiencing the wonder and majesty of the Baby-Sitters Club thanks to the glorious new Netflix show, it’s time we appreciated these iconic novels for what they are: Classics of American literature.

In fact, there will be no justice in the world until Kristy’s Great Idea is seen as a worthy rival to The Great Gatsby. Until Super Special #4: Baby-Sitters Island Adventure takes its rightful place next to Moby Dick and The Old Man and the Sea. Until Super Special #14: BSC in the USA is elevated to the status of On the Road and The U.S.A. Trilogy. Until Mystery #20: Mary Anne and the Zoo Mystery is taught alongside … um … other great American zoo mystery books!

Anyway, you get the point. Here’s what some of them might look like! (Apologies in advance if it’s still too soon to think about Claudia and the Sad Goodbye.)
sls-dev-tools Guardian — Automated best practice checks
By: Ben Ellerby
Tags: Ci, Lambda, Open Source, AWS, Serverless

I build, write and talk a lot about Serverless applications. Using AWS Serverless services extensively with my clients and, for many applications, it’s the best technology choice. Serverless adoption continues to grow, but the biggest pain-point for companies adopting Serverless is no longer vendor lock-in or cold-start times, it’s education and upskilling.

Serverless is a steep learning curve and a moving target

That is why today I’m excited to announce sls-dev-tool’s iconic CLI Dashboard interface, now called sls-dev-tools HQ, is one of several tools in the new sls-dev-tools. Today, it is joined by sls-dev-tools Guardian.

sls-dev-tools Guardian is a highly opinionated, highly configurable, automated best-practice audit tool for Serverless architectures. Like all sls-dev-tools features, it’s framework agnostic (SAM, Serverless Framework…) and can be run in one simple command.

Read the full release and how to use it today!

https://medium.com/serverless-transformation/how-to-succeed-with-serverless-automate-best-practices-2a41894721a3

sls-dev-tools is open source and built by the community for the community.

Feedback, Stars, Shares, Issues and PRs all welcome.

Ben
How to Make Your Product Likable
By: Shilpi Gupta
Tags: Product Management, Software Development, User Experience, Design, Programming

Customer Relations

I feel this is mostly ignored by developers: managing good customer relations. Please keep in mind that customers are driven more by emotion than logic. The question might appear stupid to you, but for your customer, it is like a complex problem. Your user is clueless.

Photo by Kelly Sikkema on Unsplash.

An important thing in the process is patience. You need to be patient. Users might not have enough vocabulary to express their problems or they might have reached a wrong root cause by themselves. You need to ask questions politely. Users want to be heard. Even if you agree or not, listen to them, help them to correct their understanding, and make them feel like they are the most valuable customer of all. The cumulative interaction of a customer should add to the positive emotion.

Try to delight your customers by adding some surprising and creative elements now and then. The perfect example of this is Google doodles. You sometimes just open the page out of curiosity about what’s new. And for your product, it would bring increased retention and acquisition of users.
Tips on Designing a Million Dollar Dashboard
Tags: Big Data, Analytics, Data, Data Science, UX

Tips on Designing a Million Dollar Dashboard

How to avoid the dashboard graveyard

Photo by Stephen Dawson on Unsplash

In the past few weeks, we have watched Tableau be purchased by Salesforce for 15.7 billion dollars and Looker for 2.7 billion dollars by Google.

Why are these companies so valuable?

Why are so many companies willing to pay for the services that Tableau and Looker provide?

The truth is dashboards can be expensive wastes of time and resources or they can provide insights that are worth exponentially more than the original investment. In fact, creating dashboards is a very lucrative business.

I worked for a company whose business model revolved around selling dashboards and insights to external customers. We only had four dashboard products. That was it, and yet, this company has existed for over a decade.

This is because they were really good at creating dashboards that provided impact continuously. These weren’t dashboards that you use for a month or two and forgot about. Instead, the dashboards they created were concise, clear and helped directors make decisions quickly.

They were able to do this by creating dashboards that followed a few basic principles. We wanted to share some of these basic tips below. Who knows, maybe your team will build the next million dollar dashboard.
Vue Authentication Using Sendy Auth Plugin
By: Francis Kisiara
Tags: Front End Development, JavaScript, Plugins, Vuejs

Configuring Sendy Auth

The options object includes a number of properties that can be used to configure the plugin.

driver (string)

The driver property sets the social authentication driver that will be used when performing social authentication. If a driver is not defined in the object then the app will default to using Google.

NOTE: Currently, the plugin only supports the ‘Google’ driver.

authUrl (string)

In case there is an internal authentication endpoint that is required in your application, this is where you define it. This property is also used for basic authentication that shall be covered in the second part of the series.

NOTE: For social authentication, if this is provided then the payload that will be sent to the url will be the email and password (token) obtained from the social providers.

configs

The configs property further customises the driver. It’s an object whose keys should be supported drivers only. Each definition is an object with the necessary driver configurations. For example Google’s config object expects a clientId that is used to access google apis (gapi).

configs: { google: { clientId: //enter your google api token here as a string } }

Sendy auth automatically registers two components in the global Vue instance:- the social auth and the basic auth component.
Bugs in Trust
By: Michael Boufford
Tags: Quality Assurance, Customer Service, Software Engineering, Software Development, Testing

While some software teams have instituted expensive zero-tolerance policies, most teams try to balance the priority of each bug against a mountain of other worthwhile demands. Both absolutist and pragmatist policies can be perfectly appropriate depending upon the nature of the software–for instance, I’d like the software that keeps my plane aloft to work 100% of the time thank you very much. But, if a gif of a panda rolling around in the grass pauses half-way through, it’s not the end of the world. For “system of record” software like Greenhouse, we’ve found our own tolerance level sits closer to “keep the plane in the air” than gif-stuttering consumer site, but we don’t bring everything to a halt if a button is 3px left of where it should be in IE10.

To bring clarity to how we prioritize our work, we have developed a new term that has found its way into our day-to-day conversations at Greenhouse, so I am writing this short article to share this term with the broader community. While we do make trade-offs in prioritizing certain issues, we have recently instituted a zero-tolerance policy for what we call trust bugs. Let’s define the term.

Trust bugs are distinct from other bugs, in that they cause the user to either:

not believe that the data as presented are correct, or not believe that an action taken will reliably occur

Bugs in this category are distinct because they don’t just cause frustration, they harm the relationship between a company and its customer. Trust bugs create skepticism every time a button is pressed, or a list appears, and this eventually leads to customer churn.

Trust is much harder earned than it is lost, so don’t let trust bugs sit in the backlog.
Shocking Research Warned of a Pandemic Decades Ago
By: Kim Mia
Tags: Covid 19, Humanity, History, Future, Pandemic

Shocking Research Warned of a Pandemic Decades Ago

History repeats itself so why weren’t we prepared?

Red Cross volunteers fighting the influenza epidemic of 1918 (Photo Credit: Getty Images/American Red Cross)

Reality is made up of circles. The swirling tides of historical events eventually swift back to our present. The fascinating duplicity of history is that it does nothing but repeat itself. Every event, every act, and every move can be predicted. Nothing is ever original and the outlook of the 2020s is living proof.

The simple irony of life is that the more we change, the more we stay the same.

The thought of predicting the future is no longer a fantasy but rather an existing possibility. But if we’ve walked down this path before, why do we keep falling into the same pits?
Why the U.S. Waited So Long to Regulate Carbon Dioxide
By: Charlee Thompson
Tags: Climate Science, Carbon Emissions, Climate Change, Policy, Environment

I am an environmental engineer who studied air quality and atmospheric science. I know I have a lot to learn, but I thought I at least knew the very basics of air quality and its relationship with climate change.

I was wrong. I knew the basics of the science. Not the policy.

I have all of this knowledge about how carbon dioxide (CO₂) chemically impacts the oceans and coral, and how crops will struggle to grow while weeds have potential to thrive. But not about the policies that regulate the environment we live with.

To kickstart my education, I figured, why not read a nonpartisan book about some of the most prevalent environmental policy challenges to date. (I’d recommend reading How the Government Got in Your Backyard by Jeff Gillman and Eric Heberlig.)

I made my way through several chapters, learning about invasive species management and the strict lawncare rules of homeowners associations, as well as the key differences between organic and conventional farming.

But when I reached the chapter on climate change, one thing shocked me: CO₂ has only been regulated at any governmental level in the United States since 2007.

Wait. How could I have not known this?

And how did that tiny molecule become the face of global climate change?
The Cancer Routine.
By: Lisa Martens
Tags: Short Story, Fiction, Cancer, Health, Healing

The Cancer Routine.

On seeking normalcy.

Photo by Ani Kolleshi on Unsplash

There was dropping off the kids, then some cleaning, some work (now done remotely), and then chemo. She brought a wide-brimmed hat with her wherever she went…her skin was now sensitive to the sun…and usually went to Starbucks after her treatment. She played a word game while she waited for the shots. One, two, three.

Sometimes they felt cold going in. She had to fill out the same form every day.

How was her pain?

Her anxiety?

How was she sleeping?

She lied and said she was fine across the board.

Truth was, she still needed to remind her husband to wear his sleep apnea mask. She still had to shake him when he stopped breathing. And what would happen if she was not there to wake him up? What would happen then?

So you see, she had to be there. She just had to.
People Won’t Live up to Your Expectations
By: Sudipto Chanda
Tags: Communication, People, Emotional Intelligence, Behavior, Expectations

People Won’t Live up to Your Expectations

But you can choose not to get annoyed

Image by Ryan McGuire from Pixabay

People act strangely.

Yes, it is true. People do act strangely, and they don’t always live up to your expectations.

So you called your good friend John, he didn’t pick up the phone and didn’t return your call for one whole week. You started fretting because John always returns your calls in a few hours at the most.

You can brood for days and think of all the negative things you want — John doesn’t care, your call isn’t significant enough, what the hell does he think of himself, etc. etc. Ok, this is not what you do (but many people do) when somebody doesn’t return their calls.

I’m sure there are many occasions when someone doesn’t live up to your expectations, and you get annoyed. You feel hurt, rejected, angry, upset, or some similar negative emotion.

To complete the story, John did call you a few days later and explained that he had to jump onto a plane on an hour’s notice because of his crazy boss.
There are no safe states.
By: Ben Edwards
Tags: Society, Trump, Politics

It doesn’t matter if you’re in a swing state, or a “safe” state — blue or red.

No state is safe from the ideas espoused by Donald Trump — ideas that are endorsed by the KKK — ideas that are rejected by real Republicans — ideas that are laced with cynicism, fear, bigotry, and hate.

Vote. Please. Proclaim that America has no place for these ideas*.

* It is crucial that you vote tomorrow. After the election, we’ll figure out what motivated many Americans to overlook these ideas and cast their vote for a flawed human anyway. We need to fix what allowed this. And we will.
An 18 Minute Routine for Success
By: Ashley Richmond
Tags: Health, Lifestyle, Self Improvement, Advice, Habits

The routine

Morning — 5 minutes

Taking 5 minutes in the morning to plan your day is incredibly important. Bregman recommends sitting down with your to-do list and figuring out what you can do that will make today highly successful. What progress can you make towards your goals? Where can you schedule the things on your list? Plan what you are going to do today.

“Effectively navigating a day is the same as effectively navigating down a rocky precipice on a mountain bike. We need to look ahead. Plan the route. And then follow through.”

For me, I follow a more Stoic practice — I visualize the day ahead and consider what challenges I may face. I then decide how I will deal with those challenges. I also write one thing I am grateful for, my intention for how I want to live this day, and the one thing I need to do today to be able to say that this day was a success.

These 5 minutes in the morning can look like anything. The point is to take some time to reflect on the day ahead and plan your direction.

Each hour — 1 minute x 8

Set an alarm for each hour. When your alarm goes off, take one minute to reflect.

Take a deep breath

Reflect on the productivity of the last hour — did you do what you wanted to get done? How was your focus? How are you feeling?

Set an intention for what you are going to do for the next hour. Bregman says, “Reconnect with the outcome you’re trying to achieve, not just the things you’re doing.”

“Manage your day hour by hour. Don’t let the hours manage you.” — Peter Bregman

Evening — 5 minutes

Take 5 minutes at the end of your work day to reflect on how the day went. Bregman suggests asking yourself the following questions:

How did the day go? What success did I experience? What challenges did I endure? What did I learn today? About myself? About others? What do I plan to do — differently or the same — tomorrow? Whom did I interact with? Anyone I need to update? Thank? Ask a question of? Share feedback with?

Again, mine looks slightly different. I recount the top 3 wins of the day, what I am grateful for, what I am looking forward to tomorrow, and my one thing for tomorrow. If I’m feeling particularly motivated, I will run through the questions Seneca asked himself each evening:

What did I do badly? What did I do well? How can I be better tomorrow and what tasks were left undone?

Figure out what will be most valuable to you in your evening reflection. This will be highly personal and will depend on what your work day looks like and what your goals are. Again, the point is to reflect and ensure you’re on the right path, as well as setting you up for a productive day tomorrow.