fabiochiu/medium-articles huggingface dataset:
Am I Learning Too Fast?: Reinforcement Learning in My Life
Tags: Psychology, Self Discovery, AI, Learning, Reinforcement Learning

Photo by Alex Knight on Unsplash

Reinforcement learning ties together two passions of mine, AI and psychology. I have many memories of sitting in my machine learning and computational neuroscience classes thinking about how I could apply what I was learning about robots to my own life. For those of you who aren’t familiar, reinforcement is basically a fancy term for learning from experience. A person or robot changes their behavior based on the outcomes of experiments, doing more of what produces favorable results and less or what produces unfavorable ones.

Every person or robot changes their behavior at a different rate. Some, like me, make drastic changes in response to each experience, enabling us to progress faster than more conservative learners when we get lucky and stumble upon a sample of experiences representative of our human or computational world. The downfall of this approach is the possibility of betting too aggressively based on a biased sample.

American society romanticized those who risk it all based on a hunch from entrepreneurs to politicians, when they’re right anyway. We don’t hear nearly as much about equally informative tales of misjudgment. As a 20-something early professional, I feel pressured to find a career, mission, and lifestyle and stick with them to gain an advantage in the rat race of my choice, yet I often find myself questioning my passions as I learn more about myself and the world.

TV host, author and speaker, Mel Robbins, recommends doing what energizes you rather than looking for a single passion. I found reassurance that it’s natural for priorities and interests to change over time in a video on her YouTube channel. Her words reminded me that my journey of self discovery is productive in itself and something I should relish rather than rushing through it, a realization that’s empowered me to develop a fulfilling balance between action and knowledge seeking rather than sacrificing learning for immediate productivity.

For more information on reinforcement learning and its applications in technology and life, check out my friend’s blog.

Call to action: What are some judgments you might’ve made too fast? What are some sources, conversations, and experiences you can pursue to develop a more holistic perspective?
MedMaster. By: Cyra
By: Cyra Aggarwal
Tags: Supervised Learning, Healthcare, Machine Learning, Artificial Intelligence, Reinforcement Learning

By: Cyra, Milena, Sufia

What is MedMaster?

MedMaster is a company that uses AI to predict and track children’s health through a wristband and an app.

MMband (wristband)

The MMband is a beneficial wristband for your child to wear. Not only does the MMband track and predict your child’s health, but it also ensures that your child is healthy for future days to come by providing recommendations based on the health of your child in previous days.

The MMband has many features that are all designed to ensure the health of your child. The features of the MMband include:

A monthly goal list

Stopwatch/alarm/timer

Health notifications

A watch

Customizable (child’s name, age, disability)

What data does the MMband collect?

The MMband is able to:

Count your steps each day

Predict future health problems and recommends the care needed

Detect diseases or symptoms

Measure your body temperature

Determine the BPM (beats per minutes)

Read blood flow and blood pressure

How does the band use the data? After collecting the data from the child, the A.I in the band is able to tell if everything is ok with the child. If the child is sick, the wristband can alert the parent immediately.

MedMaster App:

When you get the MMband, it also comes with an app. The app is for storing the data that the wristband collected. The MedMaster app includes:

Daily & weekly reports of his/her health progress

Reports that are compared from previous days

An option for your child to rate their day

One game for your child to be active

Parental controls and options to customize the MMband

Graphs/Charts that show how your child has progressed

Disabilities and Health Conditions

The MMband ensures that your child will be benefited and healthier for more days to come no matter what health conditions or disabilities they are experiencing. The MMband is customizable to any of your child’s needs and likings.
Sharing the Dream: AWS Deepracer Car
By: Arief Ramadhan
Tags: AWS, Deepracer, Machine Learning, Artificial Intelligence, Reinforcement Learning

In 1935, David H. Keller wrote a short story that summarized the dream of an autonomous car:

“Old people began to cross the continent in their own cars. Young people found the driverless car admirable for petting. The blind for the first time were safe. Parents found they could more safely send their children to school in the new car than in the old cars with a chauffeur.” — The Living Machine (1935) by David H. Keller

The dream, as we know it, is not far from our current reality. For instance, Volvo introduced its full autonomous test vehicle in 2017 and plans to bring its unsupervised autonomous vehicle (AV) by 2021. Google’s Waymo, which started its journey in 2009, has completed 20 million driving miles. In 2014, Tesla announced that its car would be capable of self-driving about 90% of the time. Today, all Tesla models are equipped with self-driving capability.

The dream lingers on, but not everyone believes in it. An article from Wired in 2017 titled “No one knows what a self-driving car is, and it is becoming a problem,” and another one from the Verge in March 2020 indicates that “Americans still don’t trust self-driving car.”

There is a gap between the dream and the public understanding. This gap is caused by the auto industry, which does a terrible job convincing the public how the new systems work.

The AWS Deepracer Car (Source: https://www.kainos.com/driven-by-reinforcement-learning-a-day-at-aicamp)

The best answer to this problem came from Amazon Web Services (AWS) in 2018. AWS introduced a 1/18th-scale vehicle called DeepRacer. This miniature car comes with a stereo camera, a dual-core Intel processor, and a Lidar sensor, making it capable of solving various tasks.

The Deepracer is designed to make it easier for programmers to get started with reinforcement learning. Reinforcement-learning is a set of algorithms that pick up skills through trial and error using feedback from its actions and experiences.

Matt Wood, an executive at AWS, hopes DeepRacer will help coders get a feel for reinforcement learning. He wishes that programmers would then apply reinforcement learning to industrial cases, such as wind turbine optimization or container scheduling in ports.

To play with DeepRacer, we must first train the code in a virtual world created by Amazon. We must write algorithms that reward the vehicle when they do something right, such as winning a race or avoiding an obstacle.

This year, Ray Goh from Singapore won the F1 ProAM Deepracer tournament, which featured Daniel Riccardo, the F1 racer. There is an inspiring video titled “Racing to a Dream” from Ray about his journey of winning the competition.

I do not know what “dream” in that title means, but it somehow reminds me of David Keller’s words about autonomous cars from 1935. Somehow his dream remains. With our technological advancement and the Deepracer growing popularity, I hope many more people would share the dream of autonomous cars.
The Tower of Babel in Data Science: Why We’re Out of “Control”
By: Galit Shmueli
Tags: Data Science, Control, Artificial Intelligence, Reinforcement Learning, Statistics

Metropolis-18: The New Tower of Babel (public domain — Flicker)

‘Control’ means different things in different data science communities

In his recent book Human Compatible, leading AI researcher and computer scientist Stuart Russell describes the dangers of “overly intelligent” algorithms, and proposes a solution based on incorporating uncertainty into the algorithms’ “understanding” of human preferences. Russell is puzzled how statisticians, control theory researchers, and operations researchers haven’t thought of this:

“In all the work on utility maximization, and loss function, the reward function, and the loss function are known perfectly. How could this be? How could the AI community (and the control theory, operations research, and statistics communities) have such a huge blind spot for so long, even while embracing uncertainty in all other aspects of decision making?” (p. 176)

Book cover (source: Wikipedia)

In my recent work ‘Improving’ prediction of human behavior using behavior modification I stumbled upon a similar arid land in trying to use statistical notation to describe the combination of two operations used by digital platform: prediction and modification. The lack of language puzzled me. Furthermore, my work has received surprised reactions by statisticians.

Why the surprise?

The answer has to do with the term ‘control’ — what it means to the different communities, and the role of control in algorithms/models.

Let me share my insights about the meaning of ‘control’ to different research communities, from my journey as a trained statistician, collaborating with researchers in social science, human-computer interaction, and machine learning, while also keeping an eye on what operations research and industrial engineering colleagues are talking about.
Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways
By: Mile Mirchevski
Tags: Reinforcement Learning, Autonomous Driving

The article named Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways , prepared by

Branka Mirchevska1;2, Maria Huegle1, Gabriel Kalweit1, Moritz Werling2, Joschka Boedecker1;3 , was launched today on arXive.

1.Dept. of Computer Science, University of Freiburg, Germany.

2.BMW Group, Unterschleissheim, Germany.

3.Cluster of Excellence BrainLinks-BrainTools, Freiburg, Germany.

Just my short comments about the article: very professional,

it deals with a topic that is very scientifically advanced, which is very relevant today on a global scale. I can only congratulate Branka and her colleagues for this kind of work.

The link to the article: https://arxiv.org/abs/2012.03234

The link to the video:https://www.youtube.com/watch?v=8Apmk3K9pkM&feature=youtu.be
How to train your Machine Learning dragon: Reinforcement Learning
By: Ai Explained
Tags: Machine Learning Python, Robotics Automation, Machine Learning Ai, Machine Learning, Reinforcement Learning

Branches of Machine Learning: Reinforcement Learning is the new branch

Historically, we used to speak about two types of Machine Learning: Supervised and Unsupervised Learning. We say that Supervised Learning is the one where we know historical answers of what we want to predict also in the future, whereas Unsupervised Learning is when we are not sure about what the output that we target to predict is.

But, what is the third wheel that recently appeared as a new branch that we cannot attribute to neither of the two known ones? Within the ever-growing kingdom of Machine Learning model, we now encounter models that are able to interact with the environment and learn from the feedback provided.

How does training work in Reinforcement Learning?

The reinforcement learning principle is very old: it works as if you were training your dog to go out of the house to pee. When a dog would do it, you would reward it with a nice treat or a toy. That is why we say that the positive behaviour is reinforced. Same as with training your dog, in Reinforcement Learning an agent is rewarded when the choice it made is the correct one. On the contrary, when the dog is doing a wrong thing, there will be no bacon, and the same goes for agents in Reinforcement Learning: they are punished for wrong moves.

Training by experience

Who is an agent in Reinforcement Learning?

If there is an area of Machine Learning where you should imagine a robot, then it is in Reinforcement Learning. In the middle of the puzzle that we imagine when we say “interaction with the environment”, you can visualize a robot that in RL we call an agent. There is no training dataset in Reinforcement Learning: the agent learns by experience.

Where do we use Reinforcement Learning?

Reinforcement Learning is still in the phase of expansion, but one of the most famous historical examples of this principle is AlphaGo. Other areas where we will benefit from RL are Robotics in motion control and aircontrol, manufacturing, and different areas where strategic decisions need to be made based on a sequence of events.

Keep an eye on Reinforcement Learning: the future is coming.
Thompson Sampling for Multi-Armed Bandit Problem
By: Amit Ranjan
Tags: Thompson Sampling, Reinforcement Learning, Multi Armed Bandit, Machine Learning

In this article we will see about Thompson Sampling, it’s algorithm and implementation. There will be also some discussion on Beta Distribution used in Thompson Sampling algorithm.

Multi-Armed Bandit Problem In a Nut Shell

Multi-Armed Bandit Problem is a classical problem in Reinforcement Learning where a gambler walks into casino and gamble his money on multiple single armed slot machines. He tries to explore the machines initially to find best reward giving machine and after some initial round he exploits the machine that gives maximum rewards.

Thompson Sampling

Thompson Sampling is a randomized algorithm which is most effectively used to solve this classical Multi-Armed Bandit Problem. It achieves exploration by selecting the machine with best sampled mean (generated from beta distribution) among other machines. The beta distribution becomes more and more concentrated around the empirical mean as the number of rounds of gambling increases.

If you didn’t understand much then just remember that the basic idea is to choose a machine according to its probability of being the best reward giving machine.

If you already know the Beta Distribution, you can skip that part otherwise go along.
Reinforced Learning (and Q Learning in Twin Delay DDPG)
By: Victor Yeo
Tags: Actor Critic, Deep Q Learning, Machine Learning, Reinforcement Learning, Neural Networks

This writing attempts to give some background information about Reinforced Learning and Q Learning in Twin Delay DDPG. Reinforced Learning (RL) is the principle of learning something by trial and error. In RL, the agent tries to obtain the maximum reward over a certain amount of iterations (or time).

Q Learning illustrates the reinforced learning process. The Q stands for quality. It represents the reward from the combination of action + state when the agent takes each step in the maze. In the drawing below, when approaching the checker flag, the Q value increases. When approaching the fire, the Q value decreases.

An important concept in Q learning is the temporal difference. The first two terms are the target Q value. The last term is the predicted Q value.

Deep Q learning is a method to approximate the expected return. It is only applicable in discrete action space. It works by prediction the Q value as close as possible to the target Q value, R(s, a) + gamma*max(Q(s, a)). In other words, it seek to minimise the loss between the prediction and the target. The technique to reduce the loss is by back-propagation the loss to the neural network via stochastic gradient descent (SGD).

Policy Gradient is a method to maximise expected return by directly update the weights of the neural network. The gradient of the expected return is computed w.r.t phi. Then the policy parameter phi is updated through gradient ascent.

Update the policy parameter phi by the gradient and the learning rate alpha.

In the Actor Critic model, the policy parameter of the Actor is updated through gradient ascent. The Critic model outputs a Q value which get closer to the target Q value (to approximate the expected return). The expected return is used to perform the gradient ascent to update the policy parameter. The Actor Critic model is proposed in the academic paper “Addressing Function Approximation Error in Actor Critic Models”.

The Twin Delay DDPG (TD3) model works in continuous action space. DDPG stands for deep deterministic policy gradient. “Deep” because it uses deep neural network for both Actor and Critic. It combines ideas from policy gradient and deep Q learning. It uses two target network because that adds stability to the training process.

In TD3 model, the actor target outputs action (a). The action (a) and state (s) are used as input to critic target. The two critic target output two Q value. Take the minimum value of the two Q value, add a stochastic noise to it, and we get the target Q value. The target Q value is compared to the two Q value from the critic model. From there, we get two loss value.

For Q learning in TD3 model, we use back propagation to minimise the loss value.
Transfer and Multi-Task Learning
By: Ivan Lee
Tags: Robotics, Multi Task Learning, Simulation, Reinforcement Learning, Transfer Learning

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
The Multi-Armed Bandit Problem in Reinforcement Learning
By: Amit Ranjan
Tags: Machine Learning, Reinforcement Learning, Multi Armed Bandit, Data Science

Some strategies in Multi-Armed Bandit Problem

Suppose you have 100 nickel coins with you and you have to maximize the return on investment on 5 of these slot machines. Assuming there is only one machine out of 5 which will give you maximum profit.

Strategy 1: Explore only

In this strategy what you will do is you will divide 20 coins for each of the slot machines and play with them. This might be the wise approach but it will not give you the maximum profit since out of 5 only one will give maximum profit and others might be very less.

Strategy 2: Exploit only

In this strategy you will try on all the 5 machines once and will continue on that machine where you find the maximum profit with rest of the coins. But here is the catch. Suppose at first time you get the maximum profit on Machine 1 but the Machine 5 can give maximum profit in long run. Then with this strategy you would be investing in Machine 1 itself and not getting the maximum return as you might get in Machine 5.

Strategy 3: ε — Greedy Method

In this strategy we start by setting some lower value of ε (epsilon) like 5% or 10% something like that. It says that at any time there is 5% chance that we are going to randomly pick a slot machine. There is 95% chance that at any given time we are going to play on that slot machine which has historically given us the maximum profit so far.

Let’s say at the end of 20th coin that is on 21st coin we run our random number generator and we find that based on 5% chance we should be exploiting our current knowledge today. So, what we do is that we look through the first 20th coin experience and find which machine has given us the best profit and we play on that slot machine.

Now let’s say its 22nd coin and we again look at our random generator and find that at this time we take 1/5th chance of visiting any of the 5 slot machines to purely get another machine knowledge. So that we become even more sure of our decision when the next exploit day comes.

The performance is going to depend on the choice of epsilon ε itself. Hence this will be better strategy in comparison to Strategy 1 and Strategy 2.

There can be many strategies like UCB method, Thompson Sampling etc. These are the better strategies which we will be see in another article.

This article is purely on Multi-Armed Bandits Problem!

References:
Hey!
By: Shweta Goyal
Tags: Artificial Intelligence, Reinforcement Learning, Deep Learning

Hey!

I really enjoy working with Pytorch and this framework seems to be really interesting. But I think it is more specific rather than general.

And how much time it took to train your model?
体育运动员应该先打新冠疫苗么？
By: Jiayu Liu
Tags: Dithering, Reinforcement Learning, Covid

Hi there! I’m Jiayu Liu, currently an engineering manager at Airbnb China, located in Beijing.
Soft Actor-Critic Reinforcement Learning algorithm
By: Dhanoop Karunakaran
Tags: Robotics, Software Development, Machine Learning, Artificial Intelligence, Reinforcement Learning

A general representation of reinforcement learning one-time step process. Source: [1]

Soft Actor-Critic(SAC) is one of the states of the art reinforcement learning algorithm developed jointly by UC Berkely and Google[2]. It is considered as one of the most efficient RL algorithms to apply in real-world robotics[2].

The algorithm is based on a maximum entropy RL where the objective is to find the optimal policy that maximises the expected long term reward and long term entropy[2][3].

The objective function of Maximum entropy RL. Source[3]

Read this article to know more about Maximum entropy RL

Actor-Critic architecture. Source: [4]

As the name suggests SAC is an actor-critic algorithm as shown in the above figure. In a simple description, actor-critic is a combination of policy-based and value-based approach.

If you want to know more about actor-critic, read this article

Learning of the actor is based policy gradient approach and critic is learned in value-based fashion. In SAC, there are three networks: the first network represents state-value(V) parameterised by ψ, the second one is a policy function that parameterised by ϕ, and the last one represents soft q function parameterised by θ.

Learning of the state-value function, Vψ

The learning of the state-value function by minimising the minimize the squared residual error as shown below:

Source: [6]

It means the learning of the state-value function, v is done by minimising the squared difference between the prediction of the value network and expected prediction of Q function with the entropy of the policy, π[5].

When we take the gradient or first-order derivative of the above equation, we can get the below equation:

Gradient objective of the state-value function, Vψ. Source: [6]

Learning of Q function, Qθ

Training of the Q function is based on the below equation:

Source: [6]

Where

Source: [6]

In simple words, training is done by minimizing the squared difference between predicted Q value and reward plus the discounted expectation of state-value of next state. The state value function is same as Vψ that explained above.

For enabling gradient-based learning, we take the first-order derivative of the above equation:

Grad objective of the Q value function. Source: [6]

Learning of policy, πϕ

Finally, policy network learning is based on the below equation:

.Source: [6]

Where

It’s a parameterising technique to make sure sampling from the policy is differentiable. ϵ is t is an input noise vector, sampled from some fixed distribution, such as a spherical Gaussian. Source: [6]

Where we take the gradient, we can get the below equation:

Grad objective function of policy function π. Source: [6]

As you can see in typical actor-critic architecture, policy learning depends on the Q-value to reduce the variance.

If you like my write up, follow me on Github, Linkedin, and/or Medium profile.
Aligning The Pendulum
By: Fox Beyer
Tags: Consequences, Reinforcement Learning, Relationships, Balanced Life

Balance- equipoise between contrasting, opposing, or interacting elements.

Weekly dose of mind medicine:

Value relationships. They lead to results. #ENTHUSIASM A healthy balance between consequences and reinforcement is key to a relationship based on trust. #ENTHUSIASM Find someone who listens to your situation, kills the weeds, and waters the grass. #ENTHUSIASM

Enthusiasm is the core component that drives success. Want to learn and obtain enthusiasm for yourself? I’ve created a 5 step process for you to adapt and maintain enthusiasm, called F.A.C.T.S.

Purchase your copy of Adapt And Maintain Enthusiasm In Adversity

You’ll be ready when the $h*t hits the fan.
Introduction to Reinforcement Learning. Part 5: Policy Gradient Algorithms
By: Markel Sanz Ausin
Tags: TensorFlow, Neural Networks, AI, Reinforcement Learning, Policy Gradient

In this part we will slightly change topics to focus on another family of Reinforcement Learning algorithms: Policy Gradient Algorithms [1]. We will mathematically derive the policy gradient and build an algorithm that uses it.

Among the different ways to classify the Reinforcement Learning algorithms we have mentioned so far, we still haven’t described one of the families yet. These algorithms can be grouped into Value-Based algorithms and Policy-Based algorithms.

The algorithms that only use a value or action-value function and do not implement an explicit policy are in the value-based family. These algorithms do not tell you which action you should take explicitly. Instead, they show you how much reward you will collect from each state or state-action pair. Therefore, you must choose what action to take after seeing those values. An example of this could be to always take the action with the highest Q-value. Another one could be to follow an ε-greedy policy. Among the value-based algorithms we can find Q-Learning, DQN and the other algorithms seem up to this point.

The algorithms that explicitly implement a policy function which decides which action to take in each step form the policy-based algorithm family.

These algorithms define a policy function, which decides the probabilities of taking each action from each state.

This function does not tell us how much reward the agent will receive from each state. It learns neither a value function V(s) nor an action-value function Q(s, a). These algorithms define only a policy function π(a|s), which estimates the probability of taking each action from each state.

Comparison of value-based and policy-based algorithms [1]

Advantages of value-based algorithms:

1. They can be trained off-policy more easily.

2. Significantly better sample efficiency.

3. Lower variance.

1. They can be trained off-policy more easily. 2. Significantly better sample efficiency. 3. Lower variance. Advantages of policy-based algorithms:

1. They can represent continuous actions, so they are more adequate for stochastic environments or for environments with continuous or high-dimensional actions.

2. They optimize directly the function that we wish to optimize: the policy.

3. Faster convergence.

Actor-Critic Algorithms

There exists a third group of algorithms. It tries to combine the best of both worlds: actor-critic algorithms. To achieve that, they use both a value function and a policy function. We will talk more about this family of algorithms in the future, but here’s a figure of how the different families fit together.
From data to business story: overview book about practice of Reinforcement Learning
By: Marina Maslenkova
Tags: Reinforcement Learning, Q Learning, Practice

Thomson sampling it is another way to make A/B testing (the most common way to do it is Random sampling). It is based on updating parameters of Beta distributions of each strategy, which depends on number of 1 and 0 rewards.

Q-learning is the basis of RL. Q-value describes each couple ща action and state. Temporal difference is the difference between (current reward+future Q-values with discount factor) and current Q-value. Updating Q-values comes from Bellman equation: Current Q-value is the sum of previous Q-value and current TD with learning rate.

Deep Q-learning (DQN) is more advanced method to predict Q-values with NN (for example, ANN), not simple Bellman equation. Inputs, which are states in the form of 1D vector, are forward-propagated from each input to neurons of hidden layer. Then activation function (threshold, sigmoid, rectifier) is applied and. finally the outcome, which is the vector of actions and probabilities. With Softmax method we can get the Q-value and choose the next action. Then it computes the loss error, which is back-propagated. In each neuron gradient descent updates the weights to reduce loss. Loss in case of DQN is. ½*TD^2. The whole process repeats many times. Memory of experience replay – is a number of transition the ANN can remembers to update weights.

CNN is used when your states are images. The process is similar to CNN, but there are three additional steps at the beginning: Convolution, Max Pooling and Flattering.
Reinforcement Learning Digest Part 3: SARSA & Q-learning
By: Ahmed El-Khouly
Tags: Reinforcement Learning, Temporal Difference, Q Learning, Sarsa

In the last article I have explained generalized policy iteration process and described our first reinforcement learning algorithm: Mote Carlo. In this article we will discuss the drawbacks of Monte Carlo and explore two other algorithms that can help the agent overcome shortcomings of Monte Carlo.

Monte Carlo algorithm learns from complete episodes. This can have the following drwabacks:

Monte Carlo cannot be used for continuous tasks.

Monte Carlo can be very slow for environments with long episodes.

SARSA

RL algorithms that can update Q-value estimates without having to wait for complete episodes are called Temproal difference (TD). SARSA algorithm is an on-policy algorithm where Q-value is updated at every time-step where the state is not terminal. . Therefore, this learning in this algorithm happens for quintuple: S,A,R,S’,A’. The update rule for Q functions is as follows:

SARSA Q-value update rule

Remember Bellman equation for Q-function:

Therefore, the TD-Error becomes:

One last remark, updates are weighted by learning rate (α) the controls how fast do Q-function forgets old values in favour of new experiences.

SARSA Algorithm

Trying SARSA algorithm in FrozenLake-v0 environment, proved to converge much faster than Monte Carlo on-policy algorithm as it required almost order of magnitude less training episodes.

Q-Learning

Q-Learning is another Temporal Difference algorithm that update Q-value every time step. What differentiates Q-Learning from SARSA is that the policy used to generate experiences, behaviour policy, is different from the policy being improved. Specifically, behaviour policy used in Q-Learning is ϵ-Greedy, just like Monte Carlo and SARSA, but Q-value is updated using greedy policy estimating Bellman optimality equation. In other words, the Q-value update rule in Q-Learning is estimating the Q-value target to be:

and Q-value update rule therefore becomes:

Q-learning update rule

Q-learning update rule estimates Q-value function by adding maximum estimate of Q-value achievable in next state to the immediate reward from current time step. It is the usage of such greedy policy in updating the Q-value, Q-leaning algorithm is an off-policy algorithm.

Q-Learning Algorithm

Q-Learning convergence characteristics seems somewhat similar to SARSA for FrozenLake-v0 environment.

In the next article we will how can we evolve the Q-Learning algorithm into a much powerful algorithm capable of solving more complex RL tasks.
DeepMind Open-Sources Lab2D: Environmental Design for Multi-Agent RL Research
Tags: Research, Machine Learning, Artificial Intelligence, Reinforcement Learning, Deepmind

Ancient Greek philosophers mused on nature and nurture and how they inform human behaviour, and social scientists continue this debate today. A new DeepMind scalable environment simulator takes a digital approach to the question, enabling the examination of environmental factors on AI agents.

The DeepMind Lab2D platform is designed to create “2D, layered, discrete ‘grid-world’ environments in which pieces akin to chess pieces move around.” It targets the specific needs of multi-agent deep reinforcement learning (RL) researchers, and has been open-sourced.

The DeepMind team chose 2D environments as they are inherently easier to understand than 3D environments, with “very little, if any, loss of expressiveness.” The trim format enables researchers to capture AI problems’ relevant complexity without considering the continuous-time physical environment. Moreover, 2D environments are more comfortable to design and require significantly fewer resources to run. In other words, the researchers can complete intensive calculations and tasks of model and algorithm training on standard CPUs, with advanced GPUs or specialized hardware not required.

To demonstrate the Lab2D system’s support for multiple simultaneous agents interacting in the same environment, the team used an imperfect information game called Running With Scissors.

The researchers explain that agents “must decide how to ‘play rock’ (or paper, or scissors), in addition to deciding that they should do so.” To increase rewards, agents have to identify other agents and what resources they are collecting (e.g., rocks) and secure the resource corresponding to its winning counter-strategy (e.g., paper). There also exists the possibility of agents observing their opponent’s policy development and taking countermeasures, which induces the potential for “feinting strategies” that could not be easily captured in a classical matrix game formulation.

The paper shows how 2D environment simulations require much fewer resources compared with 3D environments. The researchers had two random agents compete in Running With Scissors on a full RGB size of 80x80px, training with an average of 250,000 frames per sec on a single core Intel Xeon CPU at 3.7GHz, and the cost of running the simulation was nearly negligible.

DeepMind stresses that progress toward artificial general intelligence will require “robust simulation platforms to enable in silico exploration of agent learning, skill acquisition, and careful measurement.” To this end, it has open-sourced Lab2D to facilitate the design of multi-agent learning and intelligence tests.

The paper DeepMind Lab2D is on arXiv, and the code is available on the project GitHub.
Introduction to AI and Reinforcement Learning
By: Imran Iftikar
Tags: Artificial Intelligence, Reinforcement Learning, Technology

Introduction:

Recently, I’ve been going down an AI Rabbit Hole. It started out by just watching Youtube videos and reading articles, but eventually, I came accross a facet of AI that really, really intrigued, not just because of how interesting the concept is, but also because of the potential uses it may have. It’s known as Reinforcement Learning.

Overview of Artificial Intelligence:

To start, we need to understand what exactly Artificial Intelligence (AI) is. AI is a pretty blanket term — it’s the science of making intelligent machines, especially computer programs. We’re going to go over a subset of AI, machine learning.

Machine learning is basically using a computer system to make accurate and reliable predictions/readings/interpretations of data that the system has been given. The first two types of machine learning were supervised learning and unsupervised learning. (*This is simply an overview of each system — I don’t go too in depth when it comes to the tech used for such methods)

Supervised Learning

Supervised learning is a task-driven form of machine learning. Labelled data is used to train the system, and when new data is presented, the system will be able to make a prediction.

For example, you can feed a system images of cows and chickens, and over time, it will be able to make distinctions. The system creates a function on a graph based on the data in order to predict what output a given input will lead to — you can learn more about how it actually works here. Supervised learning is used for pattern recognition, spam detection, object recognition, etc.

Unsupervised Learning

Unsupervised learning, unlike supervised learning, is data-driven. The system isn’t set out to recognise anything, but to observe. In unsupervised learning, the system is given raw data, and needs to figure out patterns in the data. Essentially, it finds patterns in data and groups said data accordingly.

For example, say there are five red data points and five green data points. The system will recognize that the data points differ in color, and will group them accordingly. Say we add five blue points, and give each of the now 15 points a random number from 1–3. The system will recognize all of these patterns.

Unsupervised learning is mainly for clustering problems, or grouping data. It is also used for anomaly detection — that is, noticing inconsistencies in datasets. This is especially useful for things such as bank transactions.

These were the first two types of machine learning, and they’re both well and good, and pretty useful — but, as Grand Master Yoda said, “There is another.” The third type of machine learning, and perhaps the most interesting (to me, at least), Reinforcement Learning.

Introduction to Reinforcement Learning:
Reinforcement Learning In a Easy Way
By: Raghav Rathi
Tags: Artificialintelligence, Machinelearning, Reinforcementlearning

RoboticArm

From the amazing results at vintage Atari Games, DeepMind’s victory with AlphaGo, stunning breakthroughs in Robotic ArmManipulation and even beating professional players at 1v1 games like DOTA, the field of Reinforcement Learning has literally exploded in recent years. Ever since the impressive breakthrough on the image net classification challenge in 2012, the success of supervised Deep Learning have continued to pile up and people from many backgrounds have started using deep neural nets to solve a wide range of new tasks including how to learn intelligent behavior in complex dynamic environments. In this blog I will try to cover Reinforcement Learning in a easy way.

Let's start with basics, When you are doing reinforcement learning is that a lot of it takes place as a conversation between an agent and an environment.
Farewell, Udacity!
By: David Silver
Tags: Self Driving Cars, Udacity

After four and a half years, today is my last day at Udacity. On Monday, I will return to my roots in core self-driving car engineering. I’m excited!

Udacity has been the most successful and fun experience of professional life. I leave with memories of amazing students, terrific colleagues, and work of which I am proud.

I am so grateful to Sebastian Thrun and the Udacity team for recruiting me here in 2016. Together we built the Self-Driving Car Engineer Nanodegree Program, which has trained thousands of autonomous vehicle engineers, along many other amazing programs an courses, ranging from artificial intelligence to data science to web development to cloud computing, and beyond.

This small collection of photos captures a few of my many wonderful experiences with this amazing company.

The 2016 SDC Pre-Launch Dashboard!

Launching at TechCrunch Disrupt 2016!

Finding Lane Lines — the first Self-Driving Car Project

Ryan Keenan building Self-Driving Car projects

The first Self-Driving Car Team Retreat in Pajaro Dunes

Meeting Udacity students in Detroit

We won the first Udaciward!

Meeting Udacity students in Tokyo

Udaciward Outing: NASCAR in Sonoma

Teaching with Lufthansa’s FlyingLab at 30,000 feet!

Filming the final video!

We finished the Self-Driving Car Engineer Nanodegree Program!

Graduation!

Brok and the team went crazy for my birthday!

Autonomous Day at the Porsche Experience Center

Interviewing Sebastian Thrun for Udacity Talks

Working with the Infosys self-driving golf cart in Mysore, India!

Filming with the Baidu Apollo team

Teaching self-driving cars at the Navimotive Conference in Ukraine!

Presenting at NIO House in Hangzhou

South by Southwest!

The last School of Autonomous Systems Team Retreat, in San Francisco!

Interviewing C++ creator Bjarne Stroustrup

Live Teaching Samples!

Curriculum Team Q4 2019 Retreat in the redwoods

The Curriculum Team escaped!

Super Chris Vasquez!

We completed Los Pollos Hermanos Employee Training!

Ask Me Anything!

Farewell Karaoke!

Udacity is full of such wonderful people! My colleagues made me an amazing farewell video :-)

I’m a little self-conscious about sharing it, because it’s hardly modest. But the video is a tour in and of itself through my time at Udacity, and it makes me so happy and proud.

If you pay attention, you can even get some hints about what I’ll be up to next ;-)
3D Point Cloud Semantic Segmentation Using Deep Learning Techniques
By: Rucha Apte
Tags: Self Driving Cars, Point Cloud, Deep Learning, Segmentation, Pointnet

Introduction

Point cloud learning has recently attracted attention due to the development of Augmented Reality / Virtual Reality and its wide applications in areas of computer vision, autonomous driving, and robotics. Deep Learning has been successfully used to solve 2D vision problems, however, the use of deep learning techniques on point clouds is still in its infancy because of the unique challenges faced for its processing. Earlier approaches in Deep Learning overcome this challenge by pre-processing the point cloud into a structured grid format at the cost of increased computational cost or loss of depth information. 3D point cloud segmentation is the process of classifying point clouds into different homogeneous regions such that the points in the same isolated and meaningful region have similar properties. 3D segmentation is a challenging task because of high redundancy, uneven sampling density, and lack of explicit structure of point cloud data. The segmentation of point clouds into foreground and background is a fundamental step in processing 3D point clouds. One can precisely determine the shape, size, and other properties of the objects in 3D data. However, segmenting objects in 3D point clouds is not a trivial task. The point cloud data is usually noisy, sparse, and unorganized. Apart from that, the sampling density of points is uneven and the surface shape can be arbitrary with no statistical distribution pattern in data. Moreover, due to limitations in 3D sensors, the background is entangled with the foreground. Additionally, it is difficult to have a deep learning model that is computationally efficient and has a low memory footprint to perform segmentation. The segmentation process is helpful for analyzing the scene in various applications like locating and recognizing objects, classification, and feature extraction. 3D point cloud segmentation can be deployed at scene level (semantic segmentation), object-level (instance segmentation), and part level (part segmentation). Semantic segmentation is a technique that detects for each pixel, the object category that it belongs to and also treats multiple objects of the same class as a single entity.

Taxonomy for 3D semantic segmentation using Point-based Methods

Given a point cloud, the goal of semantic segmentation is to separate it into several subsets according to the semantic meanings of points. The current article focuses on studying state-of-the-art semantic segmentation techniques under the point-based methods. The taxonomy for various point-based 3D semantic segmentation techniques can be given by 4 paradigms as (a) Point-wise MLP, (b)Point Convolution, (c)RNN-based, and (d) Graph-based.
Jetson - Self-Driving Toy Car (Part: 1)
By: Greg Surma
Tags: Self Driving Cars, Robotics, Machine Learning, Artificial Intelligence, Programming

Car Assembly

To build a self-driving toy car, we need hardware that would allow us to control the car’s motion. More specifically, we need to be able to control its latitude (steering) and longitude (throttle) motion.

There are a bunch of options to start with. I recommend checking nvidia’s suggestions, donkey car docs or waveshare AI kit.

I used the last option and I can recommend it, as it provides an easy way to assemble necessary hardware and control a car’s throttle with a DC motor and a car’s steering with a servo motor.

(source: author)

Sensors

After building a car that can move, we need to make it able to sense the environment. There are multiple ways of doing that, but the most straightforward approach in the context of autonomous vehicles is to use a wide-angle front-facing camera.

In the future, we could extend our sensor suite with sonar, lidar, or another camera, but as for now, let’s focus on a single wide-angle camera with 160° FOV.

Brain

Having our car being able to move and see, we need to bridge the gap between the hardware that we already built, and the software that we are going to build next.

In order to do that, we need a single-board computer (SBC). We are going to use NVIDIA’s Jetson Nano with quad-core ARM CPU, 4GB of RAM, and most importantly 128 CUDA cores GPU that would allow us to run neural networks in real-time.

Track

Finally, we need a track for our car. My solution for that is a simple gym mat and some wooden bricks. Such an approach is a cheap, easy, and most importantly, flexible way of building a car track. The necessity of building a track every time from scratch forces to make it slightly differently every time, which impedes overfitting during the training phase. Also, with small wooden bricks possibilities of track configurations are almost limitless.

(source: author)

AI Autopilot

We humans often underestimate the complexity of things that we need to do every day, and driving a car is one such example. Considering a simple case of going from point A to point B, for us humans, it would be something relatively simple. We could do it almost without paying explicit attention to the driving task itself.

Somehow, we would ‘automatically know’, that to drive we would have to

keep the car on the road keep the distance from other cars and obstacles look out for pedestrians look out for signs obey traffic rules go in the right direction etc.

These are just simplified examples of tasks that need to be taken care of in order to safely drive and a fully complete list would be much longer than the above one.
Apple wants to build its own Self-Driving Car in 2024
By: Anees Shah
Tags: Self Driving Cars, Apple Vehicle, Apple, Electric Car, Driverless Cars

Apple Self-Driving Car: The Company is working on a new type of battery that will give the vehicle more autonomy.

Apple has set itself a goal for 2024 to produce a driverless passenger vehicle that will include its own state-of-the-art battery technology, sources close to the operation, framed within the so-called Titan Project that has been underway well unevenly, since 2014, Reuters says.

This year, the company began designing its own vehicle from scratch. At one point, Apple reevaluated its goals and focused its efforts on software and electronic devices. Doug Field, an Apple veteran who had worked at Tesla, took over the project in 2018 … and laid off 190 people from the team in 2019. Since then, Apple has made enough progress that it now aspires to build a vehicle that can get to market.

These plans contrast with those of rivals such as Waymo, of Alphabet (Google’s parent company) that has built robot-taxis to transport passengers for a driverless transport service. One of the keys to Apple’s strategy lies in a new battery that could “radically” lower the cost and increase the vehicle’s range, according to a third person quoted by Reuters, who has seen Apple’s battery design. The company has declined to comment on its future plans or products.

In this regard, Apple plans to use a unique single-cell design that increases the volume of individual cells in the battery and frees up space within the battery, eliminating the bags and modules that contain the battery materials, one of the employees said. Apple’s design means that more active material can be packed inside the battery, giving the car potentially greater range. Apple is also examining battery chemistry called LFP, or lithium iron phosphate, the person said, which is less likely to overheat and therefore safer than other types of lithium-ion batteries. “It is the next level”, have assured the sources.

Manufacturing a vehicle represents a challenge for the supply chain. Even for Apple, with an immense cash flow with expertise in electronics, but not automotive. To cite one example, it took Tesla 17 years before it finally made a profit from making cars. “If there is a company on the planet that has the resources to do that, it is probably Apple. But at the same time, it is not a mobile phone, “a person who has worked on Project Titan told Reuters.

The pandemic, yes, can interfere with the plans. Two close sources have warned that pandemic-related delays could push the start of production to 2025 or beyond.