fabiochiu/medium-articles huggingface dataset:
Golang: Serverless Microservices with gin
By: Nidhi Canopas
Tags: Golang, Serverless, AWS Lambda, Aws Api Gateway, Gin

Photo by Nathana Rebouças on Unsplash

In this blog, we’ll see how to implement serverless microservices in Golang with gin, AWS Lambda, and API Gateway.

I’m considering that you have a rough idea about serverless microservices, AWS Lambda function, and API Gateway, and if not, you can read more about it here. If you don’t have an AWS account already set up, create one.

To demonstrate serverless microservices, we’ll need to setup following things.

Lambda function ( which will be compressed file of Go binary ) API Gateway

By using the following code you can run the API in both the environments in AWS and local machine as well. Let’s first create a Lambda function in Golang.

main.go

package main



import (

"fmt"

"github.com/apex/gateway"

"github.com/gin-gonic/gin"

"log"

"net/http"

"os"

)



func inLambda() bool {

if lambdaTaskRoot := os.Getenv("LAMBDA_TASK_ROOT"); lambdaTaskRoot != "" {

return true

}

return false

}



func setupRouter() *gin.Engine {



r := gin.Default()



r.GET("/hello-world", func(c *gin.Context) {

c.JSON(http.StatusOK, gin.H{"message": "hello test completed successfully"})

})



return r

}



func main() {

if inLambda() {

fmt.Println("running aws lambda in aws")

log.Fatal(gateway.ListenAndServe(":8080", setupRouter()))

} else {

fmt.Println("running aws lambda in local")

log.Fatal(http.ListenAndServe(":8080", setupRouter()))

}

}

LAMBDA_TASK_ROOT is the variable contains the path to your Lambda function code. It will be empty if you are running the project in your local environment.

inLambda() will decide whether to execute function in AWS or in local.

Have a look at here for demonstration code.

It’s time to create binary from main.go, use the below command to generate it.

GOOS=linux GOARCH=amd64 go build -o main main.go

You will find a binary named main in the project structure. To compress it execute the following command. The reason for compressing the binary file is that the AWS Lambda function only accepts files up to the size of 50MB, which normally gets exceeded in the case of a large application.

zip lambda.zip main

you will find lambda.zip created inside your project directory.

By running go run main.go you will see the server running at port 8080. you can verify lambda running in local by calling /hello-world endpoint.

Let’s create a lambda function using the AWS console. Select Go 1.x as runtime.
Alexa Skill: Find a doctor nearby
By: Paul Deepakraj Retinraj
Tags: AWS Lambda, Find A Doctor, Alexa Skills

Skill Concept:

This Alexa skill is to search and help with finding a doctor nearby for the given criteria of plan, zip code, speciality and language.

Target Audience:

People who look for getting doctor information nearby to schedule appointments.

How it works:

The user simply starts the conversation with this Alexa skill and provide all search criteria such as current location, insurance plan, speciality and language to get some doctors nearby. The Alexa skill will search based on the input search parameters and provide top doctors in the current location.

Architecture:

Technical Components:

Amazon Alexa Skill Platform Amazon Lambda Backend database and services

Dialogue Flow:

Alexa Card:
How to create AWS Lambda Function using Java
By: Rahul Chauhan
Tags: Lambda Function, AWS, AWS Lambda, Amazon, Java

Let’s write our core lambda function logic.

Open Intellij Create New Project and select Maven (as shown below)

3. Click on Next and set a project name

4. Click on finish

5. Open your pom.xml file and add the following code

6. Create new package com.aws under src -> main -> java then create a new Java class as AwsLambdaRequest.java

7. Now create another class as AwsLambdaApp.java, which will have our core logic to calculate age.

8. Open cmd and navigate to your maven project directory (having pom.xml file)

9. Now run mvn package command. This command will create .jar package for our maven project which we will use to upload to function code

NOTE: The highlighted text is the path to the jar file. In my case, the path is C:\dev_space\AWS Lambda Demo\target\AWSLambdaDemo-1.0-SNAPSHOT-shaded.jar

10. Now open your lambda function that you have created in the Amazon console and go to Function Code section. Click on Actions and select Upload a .zip or .jar file and then click on Deploy.

11. Go to Runtime settings section and click on Edit. Update the handler info as shown below

Runtime should be set as Java 8. And com.aws is the Package name that contains the AwsLambdaApp.java file and handleRequest is the method which will be called. Then click on Save.

Now we have successfully created our lambda function and also deployed it to the server. We will now test it

12. Now click on the Test button and Configure test event. Leave Event template as it is and Enter name as calculateAge.

Update the JSON as follows

{

“day”: “05”,

“month”: “11”,

“year”: “1992”

}

and click on the Create button.

Now click on the Test button and under Execution result section, you will see the age as shown bellow
How to Send a CSV File from S3 into Redshift with an AWS Lambda Function
By: Alexander Bolaño Cervantes
Tags: Python, AWS Lambda, Redshift, S3, Etl

Introduction

Nowadays is a must to automate everything and cloud jobs are not the exceptions, as Data Engineer We need to acquire the skill of move data wherever needed, if you want to know how to start facing AWS tools in your daily routine like a data professional, this post is for you.

Step By Step

After collecting data, the next step is to design an ETL in order to extract, transform and load your data before you want to move it into an analytics platform like Amazon Redshift but in this case, only We going to move data from S3 into a Redshift Cluster using for AWS free tier.

To do that, I’ve tried to approach the study case as follows :

Create an S3 bucket. Create a Redshift cluster. Connect to Redshift from DBeaver or whatever you want. Create a table in your database. Create a virtual environment in Python with dependencies needed. Create your Lambda Function. Someone uploads data to S3. Query your data.

¡¡Let’s get started !!

Later you have finished step 1 and 2 let’s to connect to our database with the help of SQL client DBeaver or whatever you want, for this We need to remember the following data from Redshift Cluster configuration:
AWS Lambda hız ve maliyet optimizasyonu
By: Yusuf Baş
Tags: State Machine, AWS Lambda, Lambda Performance, Aws Lambda Power Tuning, Step Functions

basyusuf - Overview

Arctic Code Vault Contributor Pro Dismiss Sign up for your own profile on GitHub, the best place to host code, manage…
Basic facts on Amazon Rekognition
By: Shrimali Senevirathna
Tags: Introduction, AWS, Rekognition, AWS Lambda, Amazon Web Services

Rekognition is an AWS offering in Artificial Intelligence. It is a cloud-based Software as a service. It allows to easily analyze images. It recognizes faces , landscapes and even moods of a person. It also allows you to search and compare faces. Here are some benefits you can gain by using Rekognition Service.

Easy to Use

It does not require a massive technological knowledge because it is simply uploading images to the S 3 bucket and all the other behind the scenes are handled by the engine as it provides image recognition at the push of a button. The best part is the available API s which are used to automate the processes.

Low Cost

Rekognition currently offers free tier access, up to 5,000 API calls per month. After that, you will be charged per image call according to the latest pricings.

Scalable

It provides the facility to send thousands of images in a small time to check it faster.

Below are the main services provided by the AWS rekognition.

Object Scene detection Facial Analysis Face comparison Facial rekognition Confidence scores on processed images

From object Scene detection is accurately determines what is in an image. For example, a photo of a person on a beach may contain labels such as Swimwear (objects), Beach (scene).So we can say that it simply recognizes and labels Objects and Scenes (landscapes).

In summary facial recognition, it compares the features to the provided image and finds the perfect match and in face comparison , it compares and differentiates the given image to the set of images to find out the most similar one. In that situation it provides and confidence score to get a conclusion whether the accuracy of the process is high or low. And also facial analysis can detect the mood of a person’s image.

Here is a common and a simple use case of Rekognition.

Image moderation

Image moderation is a process that provides a space for the moderators to ensure that inappropriate content is not posted. The process is clearly displayed in the above image. When uploading an image the S3 bucket object upload triggers Lambda function. Lambda calls rekognition which examines for the content. If there were no any inappropriate content detected the image is posted and can be viewed. But if any inappropriate content is detected ,the post is rejected or else it could be posted as a manual view.

It is amazing how the technologies are becoming more powerful. Not only Rekognition , there are many platforms that we can use for our future researches and for our developments in organizations and in services.
Secure AWS API Gateway Using API Key
Tags: AWS Lambda, Security, Aws Api Gateway, Api Key

Overview

There are multiple ways to secure the AWS API Gateway. I used the API Key to secure it. I integrated API Gateway with Lambda, so I will discuss things in Lambda’s context.

Pre Requisites

Go through this AWS API Gateway’s documentation to grasp the basic understanding.

Details

Create a Lambda and add a trigger:

Adding lambda trigger

2. Lambda will create an API Gateway instance, it can be seen in the API Gateway console:

API Gateway Console

3. Create a Usage Plan and associate API and its deployment stage with it. By default, a usage plan is created and attached to test-API.

Usage Plans Console

4. Create an API key and associate it with the usage plan:

API Keys Console

5. Deploy the API

API Gateway Resource Console

6. Get the API URL for API methods:

API URL

7. Access the API using the method given below:

Using Curl

curl -X POST https://3ue9rkaxii.execute-api.us-west-2.amazonaws.com/staging/test --data '{"name": "irtiza"}' -H 'x-api-key: 'KeCZKoVY5Z9330Sq22qqh744IWr7oyjH9EBGOhLd'

Using Postman

Authorization Header

Request Body

Final Thoughts

Please share your feedback about anything that can be improved or I missed. Thank you
AWS re:invent 2020 — Point sur les annonces et nouveaux services
Tags: AWS, Awsreinvent, AWS Lambda, Aws Ec2, Reinvent

New - Use Amazon EC2 Mac Instances to Build & Test macOS, iOS, iPadOS, tvOS, and watchOS Apps |…

Throughout the course of my career I have done my best to stay on top of new hardware and software. As a teenager I…
When KMS key grant is created for Lambda execution role?
By: Jackie Chen
Tags: AWS, Aws Kms, AWS Lambda

I was working on a new Lambda function a couple days ago. It took me a while to deploy it successfully via CloudFormation. It was all due to some required IAM policies were missed in the deployment role. So I had to delete the stack then create again for a couple of times.

When I tested the Lambda function, it complains that Lambda was unable to decrypt the environment variables because KMS access was denied. But I was 100% sure the Lambda execution role has the right permission to use the KMS key (aws/lambda) to decrypt.

It turns out that it was caused by the KMS key grant is missing. Normally

the grant to the KMS key for the Lambda execution role is created automatically when:

a new function is created

the execution role is changed for an existing function.

the KMS key is changed for an existing function

Please note that the grant is given to the principle of the execution role, and the principle is unique. That is to say if the role is re-created, the principle will be different even the role ARN is still the same.

In my case, as I rebuilt the stack multiple times, and at some stage the deployment re-created the role but did not trigger new KMS grant for the new role.

The fix is simple if knowing the root cause. For me, I just changed the execution role to another one, then changed it back. In this way, it triggers Lambda to create a new grant.

Reference

https://forums.aws.amazon.com/thread.jspa?threadID=281599.
Log-based monitoring for AWS Lambda
By: Taavi Rehemägi
Tags: Lambda, AWS, Serverless, Cloud Computing, AWS Lambda

Monitoring and analytics have been an issue for Serverless systems since they were invented. While it’s easy to attach an agent like NewRelic or DataDog to a server or container, function monitoring requires a different approach. Serverless applications, where logic is distributed over a large number of functions, attaching agents or wrappers leads to cost increase and development overhead. To provide insights into FaaS architectures, Dashbird collects all your CloudWatch logs and extracts meaningful and actionable metrics.

Dashbird is a service that simplifies AWS monitoring and error alerting by collecting and analyzing CloudWatch logs and translating them into actions categorized as critical, warnings, and info. On top of preconfigured error alerts, you can also set customized alerts for whatever is important for you.

The service takes advantage of the fact that Lambda functions emit logs with a lot of useful, pre-formatted information. With smart parsing, Dashbird provides time-series metrics for invocations, memory usage, durations, while also sorting and displaying invocations separately. Every piece of data is then culminated into a single dashboard that offers a bird’s eye view of your entire system and points out problematic areas. It also acts as an error-alerting system, by recognizing exceptions, runtime errors, configuration problems, and timeouts and sending an email or Slack notification in real-time.

Dashbird requires minimal effort to set up and has no cost effect on your AWS bill. Onboarding requires a delegation to your AWS account, which allows the service to get basic data about Lambda functions and collect their logs from CloudWatch. There is no additional code needed or any intrusion to your existing infrastructure.

Enhanced AWS Monitoring

Dashbird captures all things Serverless in a single view, preventing problems from going unnoticed and gives developers an understanding of the traffic trends in the system. The main page consists of time-series metrics for invocations, mean durations and memory usage, health stats, billed duration, error reports, and function-level load distribution. From there, you can drill down into any metric or locate misbehaving functions and troubleshoot issues.

Dashbird’s single function view gives you execution, duration, and memory usage graphs, along with errors, statistics, and invocation listing. You can use this insight to optimize memory usage or view the average latency, while also seeing errors and search and live tail a Lambda function. Making for debugging and development easier and one step ahead.

Error alerting
The AWS Certified Solutions Architect Associate Exam (SAA-C02) is a JOKE!
By: Jeffrey Mclaren
Tags: AWS, Saa C02, AWS Lambda, Amazon, Aws Certification

Failed my SAA-C02 exam 2 weeks ago. Got 650/1000 (passing grade is 720). Since I remember some of the questions I encountered in the exam, I googled the phrases and terms and stumbled upon this site. Shows the actual exam questions exactly word for word:

I used Neal Davis practice exams (SAA-C02) on Udemy. Just look at the question above and then Neal’s question. The first statement is different but the second and third statements are EXACTLY the SAME!

Just a rant. This concept must be covered in the actual video course. And yes, dodgy instructors (like Neal Davis) who just copy content.

Amazon, being a colossus in the IT industry, should improve their certification process and ban dodgy instructors who just copy and paste content.
Getting started with AWS Lambda container support
By: Colin Duggan
Tags: Golang, AWS Lambda, Aws Cdk

AWS recently announced support for container images for its Lambda service. I’m a fan and advocate for all things serverless so any improvements in this space are always welcome. The following paragraphs attempt to share my initial findings as well as some tips to get started with this new capability.

The first thing to point out here is that container support does not mean Lambda can run any container. Instead you have the option of using one of the AWS base images or if you’re so inclined you can roll your own Lambda compliant custom image. Functions must also follow the Lambda event model and base images must implement the Lambda Runtime API. These idiomatic requirements prevent images from being run across any service or platforms and is an important distinction to note.

Image Types

AWS base image — These images are preloaded with one of the supported language runtimes along with all the other components required to run a container image in Lambda.

Custom image — Custom images must be compatible with Lambda. This requires implementing the Lambda Runtime API. Thankfully AWS is releasing Lambda Runtime Interface Clients for all supported runtimes so you don’t have to.

Lambda Runtime Interface

The Runtime Interface implements both the Runtime and Extensions API’s. Custom runtimes use the Runtime API to receive invocation events and send data back to Lambda within the execution environment while the Extensions API can be used to integrate Lambda with tools offering a variety of services such as security, observability, and monitoring.

Lambda Runtime Interface Emulator

The Runtime Interface Emulator comes preloaded on all AWS images providing a means to test the containerized functions locally. The emulator accepts HTTP requests which it then converts to JSON events which the Lambda function can handle.
Dynamo streams aka CDC
Tags: AWS, AWS Lambda, Solution Design, Dynamodb Stream, Dynamodb

Dynamo streams is a great feature of AWS dynamo database which helps in implementing high performance event driven applications.

What are Dynamo streams?

In layman terms, stream is a continous flow. In context of Dynamo database, stream refers to flow events (data) whenever there is change recorded in state of index.

Streams Use cases

1.CDC applications (Change Data Capture)

2.Event Driven Applications

3. IOT Applications

Dynamo streams sends data as event to a lambda function. If you are familiar with lambda functions, a single lambda function can be triggered by N number of sources. It implies that we can stream data from single table or multiple tables to same lambda fucntion.

What happens when streams are enabled on Dynamo table?

When streams are enabled on Dynamo table, it keeps track of changes happened for data stored in a index.

There are following 3 types of event trigger from stream for index

INSERT-This event returns new image of data(newly created record). MODIFY -This event returns new and old image (latest state and previous state of record). REMOVE-This event returns current state of data, which is being deleted.

Optimum Settings

Dynamo streams are processed by lambda functions. Hence its important to understand right settings for lambda dynamo trigger.

Following are the key parameters which should not be left in their default value-

Batch Size- Total number of event records processed by lambda function in single execution. By default its value is 100, however choose the right number of Batch size as per your app logic. Its the largest number of record that Lambda will retrieve from your event source at the time of invocation. Starting Position- The position in the stream where AWS Lambda should start reading. There are two types LATEST and TRIM_HORIZON.

LATEST: Lambda will retrieve the latest event available in shard (partition) for a given index.

TRIM_HORIZON: Lambda will retrieve the oldest event available in shard (partition) for a given index.

3. Maximum Retry Attempts - Its one of the most important key setting which should be set to desired number. Its default value is 10000.

lets understand, why its important-

Whenever streams send data to lambda function, it expects successful execution of streamed data. In case , for some unforseen reasons if lambda function fails, it will retry for N number of times unless it executes successfully.

Addtionally, default age of any stream record (maximum_record_age_in_seconds) is 604800 seconds(7 days), which means older stream will be available for above duration and it will block sub sequnent events of particular event type(Insert/Modify/Remove) for a given index, unless that failed event proccessed successfully by lambda function.

Its important to note that the stream record age is independent of data item present in Dynamo table.

Thanks for reading the article. Please share you feedback/ queries/ corrections in comments section.

Cheers :)

Previous Artciles:

Kubernetes Architecture

Terraform Start-up guide
What is EC2 instance in aws
By: Ashish Kapure
Tags: AWS, Cloud Computing, What Is Ec2, AWS Lambda, Cloud Services

What is EC2 instance in aws

Today i’m going to be talking about what isa easy to use instance so guysec 2 stints belong in aws so guys and in this article i’m going to be talking about what is that easy to an instance so guys EC2 instance is a virtual server in amazon elastic compute cloud for running application on on the amazon web service aka aws infrastructure aws is a comprehensive involving cloud compute platform ec2is a service that allows business subscriber to run their app application programs or servers in the computer environment so guys ec2 i told you guysisaacis a platform that people run their application program or or data server

So guys ec2 is very secure resizable compute capacity in the cloud it is designed to make webs web skill cloud computing easier for developer to usei told you guys about easy to in this article so guys if you guys liked it please subscribe and i’m going to keep going on because i’m not done yet you guys if it provides you with complete control of your computing resources and let you run on amazon proven compute cloud how do we use ec2 ec2instance is used virtually it is not it is not like a real thing is aec2 is a virtual machine andec2 elastic compute ec2 instance is a virtual machine

That you can use to run the application in aws aka amazon web service when setting up ec2instance you can customize configure cpu storage memory and networking resources and then guys you guys actually can create your own instance and do any thing in the ec ec2ec2 is a very good thing to use when ever you are you’re like using aws or like aws you should use a ws to store your cloud or just use other company but like aws is a very famous company and a very secure company and the second question is is ec2 free ec2 is free for a month if you want to try it but is it but it is not free for the whole time you have to pay to to to use ec2and easy to it depends on your server how many how many like applications you are running on like you are you are you are using how many application you want to use how many honey how many websites you are running on it so guys easy to actually like save your money because it is not it is not that expensive as like other servers last question but not the least is ec2 a server yes ec2 is a certain waybye guysi hope you guys like the article. and please comment below because i read everything coming by see you later bye peace. read more information...
AWSLambda関数の解説とAPI化の手順
Tags: AWS, AWS Lambda

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Adding a Pandas Layer to AWS Lambda
By: Chris Jones
Tags: Pandas, AWS Lambda, Python, Data Science

A step by step guide to making Pandas available as a Lambda Layer in AWS …

This guide assumes a working knowledge of AWS Lambda functionality.

Why: When attempting to run Pandas in your AWS Lambda you receive the error code below.

“numpy: cannot import name ‘WinDLL’ from ‘ctypes’ (/var/lang/lib/python3.7/ctypes/__init__.py)”.

This is due to incompatibility between Windows compiled version of Pandas and AWS Linux.

The solution is to use an AWS Linux image with Docker to compile the Pandas libraries.

Working with Docker

Download docker at “https://www.docker.com/products/docker-desktop” and install using the default specs.

When installation is complete, open Docker. This will start Docker in the background. lick Show Hidden Icons on Windows Taskbar to verify that Docker is running.
Why AWS SQS trigger Lambda’s batch size should be considered for setting the Queue visibility timeout ?
By: Hanumanth Reddy Aredla
Tags: AWS, Aws Lambda Functions, Sqs Lambda, AWS Lambda, Aws Sqs

Sometimes as a developer we concentrate on complex issues and overlook simple things. Once such thing which I did recently was the Batch Size option in Lambda SQS Event mapping(SQS lambda trigger). I have always set it to default value i.e. 10 and wrote the code to handle the same(loop through each message and process one by one).

I knew the importance of the batch size and how it works but never connected how this could impact the Lambda Execution Time and SQS Visibility Timeout.

Here is a sample data to understand this:

Each column represent total time taken by lambda function to process messages

Scenario 1: Lambda takes 10 Secs to process each message.

As you can see in the screenshot with heading scenario 1 will give you time needed for lambda to process ’n’ number of messages. If the payload received by lambda has maximum possible message i.e. 10, then the total time needed for lambda execution is 100 secs.

The same goes for other scenarios too. 50 secs per each msg => Max time 500 secs.

Considering Scenario 5(with batch size as 10), even though the time need to process each message is 50 secs, the visibility timeout cannot be less than 500 Secs i.e. 8 mins 20 secs to handle worst case scenario i.e. when the lambda receives 10 messages in the batch. If the visibility timeout is less than 500 secs then the same message will reappear in the Queue and will be available for consumer for processing which result in DUPLICATE.

The reason we overlook this often is, it is very scarce we receive 10 message in a single shot. AWS says the Queue has to be bombarded with huge number of message per second to receive 10 messages(that too with no guarantee).

Now we got an understanding how important the batch size is, to set the Queue Visibility timeout and the total lambda timeout.

On a personal note, I always thought batch size of 5–6 will be better and effective which will give a max time of 2–3 minutes for processing each message.
Upload File To S3 With Only Api Gateway
By: Ufuk Dag
Tags: AWS, AWS Lambda, Aws S3, Aws Api Gateway

Today, I want to create a serverless architecture to upload a file or something else to AWS S3 without writing any line of code.

If you are trying to upload an invoice, a picture, or other things to AWS S3 and when you google it, you will see API Gateway and AWS Lambda integration to succeed. But if you don’t control any business logic in the Lambda function, It’s not a suitable solution.

First, you need to create an AWS API Gateway and choose Rest API integration to use other AWS Services (in this case S3).

Now give a name for your Api Gateway and create it.

Create a new resource and give it a name for example “save”.

Create a new Put method and choose AWS services at left side. Choose your region and select S3 for the service. If you want to add some prefix to your object, add parameters using brackets.

Add your parameters in the Request Method.

Add your parameters also in the Integration Request.

That’s it. You are ready yo use your endpoint. You can add your parameters into request and save your objects directly into S3.
If you love AWS lambda and containers You will love this
Tags: Containers, AWS Lambda, Java, Serverless Architecture

What does AWS provide for Container Image Support?

AWS provides a set of tooling to jump right into running containers.

Base images for supported languages Java, Python, Node, Go, .Net & Ruby. For other languages, AWS provides base image for custom runtimes.

for supported languages Java, Python, Node, Go, .Net & Ruby. For other languages, AWS provides base image for custom runtimes. Runtime interface clients — client libraries that implement the Runtime API . This is available for all supported programming languages.

client libraries that implement the Runtime API . This is available for all supported programming languages. Runtime Interface Emulator for performing local test of the function.

Writing a function in Java

Let’s write a traditional “Hello World” function in Java. We will use the Runtime interface client for Java by adding the library dependency.

Next, we will write the handler method in a class.

This is a basic form of handler without any event as input. A variation of this handler is to accept and return an object. This will look like

If the lambda accepts events from one or more sources, the below variation can be used.

The next step is to build this function and containerize as a docker image. We will use the base image for Java provided by AWS.

We build the code and upload the container image to an Elastic container registry(ECR) repository. Here is a simple shell script to do this.

We can create a lambda from the AWS console and choose the Container Image template and refer our ECR repository.
How to Prevent Code Injection Vulnerabilities in Serverless Applications (Part 1/2)
By: Manuel Benz
Tags: Security, Serverless, Microservices, AWS Lambda, Web App Security

Fixing the Vulnerability

Generally speaking, such injection vulnerabilities, e.g., OS Command Injection, SQL Injection, Code Injection, XSS, etc., stem from user input being used directly in sensitive operations.

To prevent such vulnerabilities, there are two solutions, first, if possible, do not use user input in sensitive operations at all, and/or, second, validate that the user input has a proper format and is not able to exploit the sensitive operation.

Do not use an OS Command for Downloading the Document

In our ServerlessGoat example, the OS Command Injection attack is possible because the document_url user input directly flows into an OS command invocation of curl . Using curl to download the MS-Word document seems to rather be a convenience-driven decision. By no means, it is necessary to use curl to download a file. Instead, we can just use some JS native way to acquire a web-resource and store it in the file system, e.g., with the request library, and afterward propose this downloaded file to the catdoc command. Using this approach, no user input will directly flow into the OS command anymore. Even more, with such a fix, proposing some bogus document_url as input will rather lead to the request API failing fast due to not being able to acquire any proper web-resource.

Note: Indeed, AWS has recently removed the curl command from their amazonlinux:2 runtime. This certainly helps to prevent OS-Command Injections like the one in ServerlessGoat since users are rather forced to use a proper library for sending web requests instead of curl .

What if I need to pass User Input to a Sensitive Operation?

In certain scenarios, one will not be able to completely prevent using user input in an OS Command, e.g., due to the need for a specific application which’s functionality cannot be mimicked by a third-party library or if the runtime-performance that only a binary application can provide is required.

In such cases, it is important to properly validate the user input before it is used in the OS command. Such validation is usually done by using either denylists, i.e., excluding inputs that contain specific characters or match a certain format, or by using allowlists, i.e., only allowing inputs with a certain format or input that matches an item in a given list of constants.

In Part 2 of this series, I will explain how to set up AWS Web Application Firewalls (WAF) to secure your serverless application from injection attacks.

The AWS WAF proposes a deny- and allowlist approach to conduct exactly such input validation for multiple functions even before they are invoked at all. 🛡
AWS Lambda Changes Announced at Re: Invent 2020
By: Thomas Gallagher
Tags: AWS, AWS Lambda, Reinvent, DevOps

As with every Re: Invent, AWS always drops a ton of news, roadmaps, and new features for their users. Today’s Re: Invent was no different, with AWS announcing some changes to their Lambda service.

Performance Capacity Changes

AWS announced today that users can now allocate up to 10GB of memory, which is close to 3 times the amount users were previously able to allocate. According to AWS’s blog post, this affords users access to up to 6vCPUs for each execution.

With these changes in capacity, AWS hopes that users will find things such as modeling, genomics, and high-performance computing to be easier to implement and scale using the Lambda service.

Furthermore, with an inverse relationship between memory allocation and duration, it is plausible that increasing memory could also slightly decrease the costs of certain functions. Anyone using Lambda will be happy to learn that their functions are more customizable after today’s Re: Invent keynote.

Per Millisecond Billing

Another change that Lambda users will be thrilled to hear about is the new billing model. Instead of having a minimum execution time, Lambda will now round up to the nearest millisecond, and there is no minimum execution time. According to the AWS blog post about the new pricing model, this change means users pay less most of the time, and it will be most noticeable when you are running functions with execution times lowers than 100ms.

AWS included a great breakdown of the math in the above blog post, which I will include below:

100,000 users * 20 invocations * 30 days = 60 million invocations. The Lambda monthly compute charges with the old 100ms rounded up pricing would have been: 60 million invocations* 100ms * 1G memory * $0.0000166667 for every GB-second = $100 With the new 1ms billing granularity, the duration costs are: 60 million invocations * 28ms * 1G memory * $0.0000166667 for every GB-second = $28

As you can see above, the new pricing model has the potential to save customers hundreds, potentially even thousands, of dollars per month, assuming they’re running multiple functions with invocations similar to the one included above.

AWS mentioned that they are activating the 1ms billing granularity in the December 2020 billing period, so Lambda users should start seeing billing benefits immediately!

Conclusion

Re: Invent is always a super exciting time for nerds because we get more and more cloud-oriented toys to play with. I am excited to see where AWS goes next, especially considering some of the other announcements today, like macOS in EC2 and a new EBS volume type.

If you enjoyed reading this article, please feel free to check out my other Medium articles, and to visit my LinkedIn, where I am active and always willing to chat about all things AWS.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬
How to use Amplify to Leverage AWS CDK with Cognito, Lambda, and API Gateway — Part 1
By: Amchelle Clendenin
Tags: Amplify, Aws Cdk, AWS Lambda, Aws Cognito, Aws Api Gateway

AWS Amplify CLI is a great and easy way to integrate with AWS cloud services at a basic level. However, to really make your development secure and easy to replicate, the AWS Cloud Development Kit (AWS CDK) is your go to framework. CDK with Amplify is a powerful combination that can be a little difficult to establish so I wanted to create a step by step guide.

If you haven’t already, create an AWS account: https://aws.amazon.com/

Create an AWS account

Got to top right corner click: “Create an AWS Account” Once, you have successfully signed in, go to the AWS IAM console and create a new user. Under Access management, click Users, then Add user.

3. Create user name and choose “Programmatic access”, then click “Next: Permissions” in the bottom right.

4. Continue creating the new user until you see your Access key ID and Secret access key. Keep this visible, we will need these for the next steps.

Set up environment

Now, let’s set up the CDK.

The AWS CDK has support for the following programming languages: TypeScript, JavaScript, Python, Java, and C#. For his tutorial I will use TypeScript.

I’m going to install the AWS CLI in order to configure our CDK credentials with AWS services for Linux. Use the following link to install on different operating systems.

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

Open a terminal and install the AWS CLI



unzip awscliv2.zip

sudo ./aws/install curl " https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip " -o "awscliv2.zip"unzip awscliv2.zipsudo ./aws/install

2. Use aws configure to set up your environment. It will require your access key ID and secret key from the previous AWS user creation step. Choose a default region (e.g. us-east-1, us-east-2, use-west-2).

aws configure

Now we are ready to create our CDK project!

Create CDK project

Install the AWS CDK Toolkit

npm install -g aws-cdk

2. Create a new directory

mkdir cdk-example && cd cdk-example

3. Create a new TypeScript CDK project

cdk init example-app --language typescript

We have now successfully set up our CDK project and can begin writing the CDK code for deployment.
AWS re:Invent 2020 — Evolution majeure de la facturation de Lambda
By: Thomas Linck
Tags: AWS, AWS Lambda, Reinvent

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
My Understanding of Serverless
By: Thomas Step
Tags: AWS, AWS Lambda, Aws Fargate

1 December 2020

by Thomas

When I first started getting into AWS, I remembered seeing and hearing posts and videos about serverless. How it could speed up applications and an entire organization, how it reduced costs, how it was a revolution. I had visions of grandeur and wonder at how Amazon managed to do all of this without a server. Fast forward a few months and I started realizing what serverless truly means. I have worked closely with Lambda functions for over a year and with Fargate for about half of a year at this point. I am by no means calling myself an expert, but at the very least I can explain what is meant by “serverless”.

Any serverless offering means that the user does not have to manage any infrastructure. The hardware is provisioned and ready to go, so after any configuration is plugged in, your code can start working. It’s a quick way to get something off the ground. The complicated operations work is already handled by the cloud provider so an engineer can focus more on code and less on maintaining a server. There is still a server though. A generic server that fits most use cases and can plug-and-play with a user’s code. The biggest, most promising part of Lambda that acts as a selling point for most organizations is the billing model.

AWS Lambda has a cult-like following. I have met people who love it and do everything they possibly can with it, and I have met people who steer clear of anything called a serverless function. No one can argue that the billing model is one of the cheapest out there for computing. With Lambda, a user pays only for when the function is being used or the duration of invocation. It is weirdly inexpensive compared to other computing services like EC2 where you have to pay for the instance even if it is not being used. I like to think of a Lambda function as a generic server that gets its logic uploaded by us. It removes all of the boilerplate that we would have to go through setting up an EC2 instance to receive traffic and call our logic. Since that’s such a common use case, AWS made it into a service and made it so fast that they could provision a new server without the client noticing it.

Fargate is another serverless technology that I have recently picked up and played with. Like Lambda, it’s a boilerplate server. Instead of the user uploading specific logic in the form of a function, we upload a Docker image for the Fargate instance to run. Fargate is a little less abstract than Lambda, but we can still reap the serverless benefits. There are no EC2 instances to patch and update, and Docker is ready to run our image as soon as we hand it off.

When it comes to serverless technology, I once thought there was some sort of magic going on, but in reality, all the infrastructure and servers still exist. The heavy operations work like provisioning and management is abstracted away from us by the cloud provider, so we are left with a quick and easy way to deploy code and serve clients. In short, serverless isn’t magic, it’s just abstraction.
AWS Lambda Layer — Simplified Dependency Management and Deployment with Java and SAM
By: Abhra Prakash Deb
Tags: Cloudformation, AWS, AWS Lambda, Maven, Java

Introduction

In the programming world, including libraries and dependencies in the function code is a common practice required for successful compilation and execution. Whether it is an SDK or external library, it is beneficial for the developers to keep the deployment package small, if possible, containing just the core function. In the Serverless domain, AWS Lambda Layer lets the developer achieve just that. It allows creating a separate archive called a layer containing additional code, such as libraries, dependencies, or custom runtimes. This post focuses on building a Layer and including it in the Lambda function using AWS Serverless Application Model template.

Lambda + Layer Deployment

Lambda Layer Use Cases

Simplified Dependency Management— Use and share common and reusable dependencies between functions in or across projects and other AWS accounts. A function can use up to 5 layers at a time.

Large Dependency Set — The application consists of a large dependency set, and a less frequent deployment is desired for those common dependencies. However, there is a hard limit of 50MB for compressed deployment package and 250 MB unzipped, including layers.

Reduce Deployment Time — Smaller package size for quicker deployment to save time and reduce the size of uploaded archives. Bear in mind, AWS Lambda has a soft limit of 75 GB storage space per region for Lambda and Layers.

Custom Runtime — The application used a language that is not currently part of the supported Lambda runtimes, and a custom runtime is required.

Lambda Function

This is a simple Lambda function handler written in Java with an S3 event trigger.

Lambda Handler

Maven Assembly

Utilize Maven assembly to create the Layer zip archive with java/lib classpath consisting only of the project’s dependencies.

Compile and Package

> mvn clean package generates the following 2 artifacts.

target/java-lambda-layer-example-1.0.0.jar, which consists of only the Lambda function code. target/java-lambda-layer-example-dependencies.zip, which is the Lambda layer dependencies zip file.

Deployment

Deploy the application using AWS SAM Template. The Lambda function refers to the layer for dependencies.

GitHub

References
Code Signing for AWS Lambda
By: Rajitha Jayawardena
Tags: AWS, AWS Lambda

Users of Lambda, the Amazon Web Services (AWS) serverless computing platform, can now ensure that only approved and unchanged code can be executed in their environments. This frees up organizations in their implementation pipelines from the responsibility of creating gatekeeper components.

Code signing is the mechanism by which executables and scripts are digitally signed to confirm the author of the program and guarantee that the code has not been changed or manipulated since it was signed. To verify validity and integrity, the system employs the use of a cryptographic hash.

Several useful features can be supported by code signing. Code signing is most widely used to provide authentication during deployment; it can also be used to help avoid namespace conflicts in certain programming languages. Almost any implementation of code signing will have some form of digital signature mechanism to verify the author’s or development system’s identity, and a checksum to verify that the object has not been changed. It can also be used to provide object versioning information or to store other object meta data.

Through the AWS Signer console, the Signer API, SAM CLI or AWS CLI, you can create digitally signed code objects using the Signing Profile.

Administrators create a Signing Profile, a resource in the AWS Signer that uses AWS Identity and Access Management (IAM) to create signatures and grant developers access to the signing profile. Inside Lambda, administrators use a new resource called Code Signing Configuration (CSC) to define the permitted signing profiles. CSC lets you define the accepted signing profiles and configure whether to alert or deny deployments if signature checks fail. To allow the code signing functionality, Code Signing Configurations can be added to individual Lambda functions. These functions are now starting to validate signatures during deployment.

CSC helps organizations to impose a terms of responsibilities between administrators and developers. CSC can be used by administrators to set code signing policies for functionality, and developers can deploy code for functions.

By adding a code signing configuration to a function, you can allow code signing for existing functions. The AWS Lambda console, Lambda API, AWS CLI, AWS CloudFormation, and AWS SAM allow you to do this. When using Code Signing for AWS Lambda, there is no extra cost. For AWS Lambda, you pay the regular price.

For more information please refer to the AWS Signer documentation.
Calling external REST Service from AWS Lambda? (Part 2)
By: Jörg Grote
Tags: Tibco, AWS Lambda, Restful Api, Oauth, JavaScript

usage of OAuth Bearer Tokens

as a sample, I use here a TIBCO Cloud Service to create a new TIBCO Cloud Live Apps Case Instance in the Case Management System.

As you can see this Service requires to use of a Bearer Token to perform the request against TIBCO Cloud.

The Token can get generated here:

TIBCO Cloud — Settings — OAuth access token generation

In AWS Lambda the API can get called like here using pure AWS SDK Core functionalities:

This Sample shows smoothly how to use OAuth, REST Type, URI definition, Input Interface handling, including validation, and how to handle the Response Data to return the newly created Case InstanceID “caseReference”