# Foundations of Computer Science/Information Representation

## Information Representation

### Introductory problem

Computers often represent colors as a red-green-blue (RGB) set of
numbers, called a \"triple\", where each of the red, green, and blue
components is an integer between 0 and 255. For example, the color (255,
0, 10) has full red, no green, and a small amount of blue. Write an
algorithm that takes as input the RGB components for a color, and
returns a message indicating the largest component or components. For
example, if the input color is (100, 255, 0), the algorithm should
output \"`Largest component(s): green`\". And if the input color is
(255, 255, 255), then the algorithm should output
\"`Largest component(s): red, green, blue`\".

### Overview of this chapter

One amazing aspect of computers is they can store so many different
types of data. Of course computers can store numbers. But unlike simple
calculators they can also store text, and they can store colors, and
images, and audio, and video, and many other types of data. And not only
can they store many different types, but they can also analyze them, and
they can transmit them to other computers. This versatility is one
reason why computers are so useful, and affect so many areas of our
lives.

To understand computers and computer science, it is important to know
something about how computers deal with different types of data. Let\'s
return to colors. How are colors stored in a computer? The introductory
problem states one way: as an RGB triple. This is not the only possible
way. RGB is just one of many color systems. For example, sometimes
colors are represented as an HSV triple: by hue, saturation, and value.
However, RGB is the most common color representation in computer
programs.

This leads to a deeper issue: how are *numbers* stored in a computer?
And why is it important anyway that we understand how numbers, and other
different types of data, are stored and processed in a computer? This
chapter deals with these and related questions. In particular, we will
look at the following:

1.  Why is this an important topic?
2.  How do computers represent numbers?
3.  How do computers represent text?
4.  How do computers represent other types of data such as images?
5.  What is the binary number system and why is it important in computer
    science?
6.  How do computers do basic operations such as addition and
    subtraction?

### Goals

Upon completing this chapter, you should be able to do the following:

1.  Be able to explain how, on the lowest level, computers represent
    both numeric and text data, as well as other types of data such as
    color data.
2.  Be able to explain and use the basic terminology in this area: bit,
    byte, megabyte, RGB triple, ASCII, etc.
3.  Be able to convert numbers and text from one representation to
    another.
4.  Be able to convert integers from one representation to another, for
    example from decimal representation to two\'s complement
    representation.
5.  Be able to add and subtract numbers written in unsigned binary or in
    two\'s complement representation.
6.  Be able to explain how the number of bits used to represent data
    affects the range and precision of the representation.
7.  Be able to explain in general how computers represent different
    types of data such as images.
8.  Be able to do calculations involving amounts of memory or download
    times for certain datasets.

### Data representation and mathematics

How is data representation related to liberal education and mathematics?
As you might guess, there is a strong connection. Computers store all
data in terms of binary (i.e., base 2) numbers. So to understand
computers it is necessary to understand binary. Moreover, you need to
understand not only binary basics, but also some of the complications
such as the \"two\'s complement\" notation discussed below.

Binary representation is important not only because it is how computers
represent data, but also because so much of computers and computing is
based on it. For example, we will see it again in the chapter on machine
organization.

### Data representation and society and technology

*The computer revolution*. That is a phrase you often hear used to
describe the many ways computers are affecting our lives. Another phrase
you might hear is the *digital revolution*. What does the digital
revolution mean?

Nowadays, many of our devices are digital. We have digital watches,
digital phones, digital radio, digital TVs, etc. However, previously
many devices were *analog*: \"data \... represented by a continuously
variable physical quantity\" [^1] Think, for example, of an old watch
with second, minute, and hour hands that moved continuously (although
very slowly for the minute and hour hands). Compare this with many
modern-day watches that shows a digital representation of the time such
as 2:03:23.

This example highlights a key difference between analog and digital
devices: analog devices rely on a continuous phenomenon and digital
devices rely on a discrete one. As a second example of this difference,
an analog radio receives audio radio broadcast signals which are
transmitted as radio *waves*, while a digital radio receives signals
which are streams of numbers.[^2]

The digital revolution refers to the many digital devices, their uses,
and their effects. These devices include not only computers, but also
other devices or systems that play a major role in our lives, such as
communication systems.

Because digital devices usually store numbers using the binary number
system, a major theme in this chapter is binary representation of data.
Binary is fundamental to computers and computer science: to understand
how computers work, and how computer scientists think, you need to
understand binary. The first part of this chapter therefore covers
binary basics. The second part then builds on the first and explains how
computers store different types of data.

## Representation basics

### Introduction

Computing is fundamentally about information processes. Each computation
is a certain manipulation of symbols, which can be done purely
mechanically (blindly). If we can represent information using symbols
and know how to process the symbols and interpret the results, we can
access valuable new information. In this section we will study
information representation in computing.

The algorithms chapters discuss ways to describe a sequence of
operations. Computer scientists use algorithms to specify *behavior* of
computers. But for these algorithms to be useful they need data, and so
computers need ways to represent data.[^3]

Information is conveyed as the
content of messages, which when interpreted and perceived by our senses,
causes certain mental responses. Information is always encoded into some
form for transmission and interpretation. We deal with information all
the time. For example, we receive information when we read a book,
listen to a story, watch a movie, or dream a dream. We give information
when we write an email, draw a picture, act in a show or give a speech.
Information is abstract but it is conveyed through concrete media. For
instance, a conversation on the phone communicates information but the
information is represented by sound waves and electronic signals along
the way.

Information is abstract/virtual and the media that carry the information
must be concrete/physical. Therefore before any information can be
processed or communicated it must be quantified/digitized: a process
that turns information into (data) representations using symbols.

People have many ways to represent even a very simple number. For
example, the number four can be represented as 4 or IV or `||||` or 2 +
2, and so on. How do computers represent numbers? (Or text? Or audio
files?)

The way computers represent and work with numbers is different from how
we do. Since early computer history, the standard has been the binary
number system. Computers \"like\" binary because it is extremely easy
for them. However, binary is not easy for humans. While most of the time
people do not need to be concerned with the internal representations
that computers use, sometimes they do.

### Why binary?

Suppose you and some friends are spending the weekend at a cabin. The
group will travel in two separate cars, and you all agree that the first
group to arrive will leave the front light on to make it easier for the
later group. When the car you are in arrives at the cabin you will be
able to tell by the light if your car arrived first. The light therefore
encodes two possibilities: on (the other group has already arrived) or
off (the other group hasn\'t arrived yet).

To convey more information you could use two lights. For example, both
off could mean the first group hasn\'t arrived yet, the first light off
and second on indicate the first group has arrived but left to get
supplies, the first on and second off that the group arrived but left to
go fishing, and both on that the group has arrived and hasn\'t left.

Note the key ideas here: a light can be on or off (we don\'t allow
different level of light, multiple colors, or other options), just two
possibilities. But the second is that if we want to represent more than
two choices we can use more lights.

This \"on or off\" idea is a powerful one. There are two and only two
distinct choices or states: on or off, 0 or 1, black or white, present
or absent, large or small, rough or smooth, etc.---all of these are
different ways of representing possibilities. One reason the two-choice
idea is so powerful is it is easier to build objects---computers,
cameras, CDs, and so on---where the data at the lowest level is in two
possible states, either a 0 or a 1.[^4]

In computer representation, a *bit* (i.e., a binary digit) can be a 0 or
a 1. A collection of bits is called a *bitstring*. A bitstring that is 8
bits long is called a *byte*. Bits and bytes are important concepts in
computer storage and data transmission, and later on we\'ll explain them
further along with some related terminology and concepts. But first we
will look at the basic question of how a computer represents numbers.

### A brief historic aside

Claude Shannon is considered the
father of information theory
because he is the first person who studied and built mathematical models
for information and communication of information. He also made many
other significant contributions to computing. His seminal paper "A
mathematical theory of communication" (1948) changed our view of
information, laying the foundation for the information age. Shannon
discovered that the fundamental unit of information is a yes or no
answer to a question or one bit with two distinct states, which can be
represented by only two symbols. He also founded the design theory of
digital computers/circuits by proving that propositions of Boolean
algebra can be used to build a \"logic machine\" capable of carrying out
general computation (manipulation of two types of symbols).
Data, another term closely related to
information, is an abstract concept of representations of information.
We will use information representations and data interchangeably.

### External and internal information representation

Information can be represented on different levels. It is helpful to
separate information representations into two categories: external
representation and internal representation. External representation is
used for communication between humans and computers. Everything we see
on a computer monitor or screen, whether it is text, image, or motion
picture, is a representation of certain information. Computers also
represent information externally using sound and other media, such as
touch pads for the blind to read text.

Internally all modern computers represent information as bits. We can
think of a bit as a digit with two possible values. Since a bit is the
fundamental unit of information it is sufficient to represent all
information. It is also the simplest representation because only two
symbols are needed to represent two distinct values. This makes it easy
to represent bits physically - any device capable of having two distinct
states works, e.g. a toggle switch. We will see later that modern
computer processors are made up of tiny switches called transistors.

### Review of the decimal number system

When bits are put together into sequences they can represent numbers. We
are familiar with representing quantities with numbers. Numbers are
concrete symbols representing abstract quantities. With ten fingers,
humans conveniently adopted the base ten (decimal) numbering system,
which requires ten different symbols. We all know decimal representation
and use it every day. For instance, the arabic numerals use 0 through 9.
Each symbol represents a power of ten depending on the position the
symbol is in.

So, for example, the number one hundred and twenty-four is
$(1\times100) + (2\times10) + (4\times1)$. We can emphasize this by
writing the powers of 10 over the digits in 124:

`10^2  10^1 10^0`\
`   1     2    4`

So if we take what we know about base 10 and apply it to base 2 we can
figure out binary. But first recall that a bit is a binary digit and a
byte is 8 bits. In this file most of the binary numbers we talk about
will be one byte long.

(Computers actually use more than one byte to represent most numbers.
For example, most numbers are actually represented using 32 bits (4
bytes) or 64 bits (8 bytes). The more bits, the more different values
you can represent: a single bit permits 2 values, 2 bits give 4 values,
3 bits gives 8 values, \..., 8 bits give 256 values, and in general *n*
bits gives $2^n$ values. However when looking at binary examples we\'ll
usually use 8 bit numbers to make the examples manageable.

This base ten system used for numbering is somewhat arbitrary. In fact,
we commonly use other base systems to represent quantities of different
nature: base 7 for days in a week, base 60 for minutes in an hour, 24
for hours in a day, 16 for ounces in a pound, and so on. It is not hard
to imagine base 2 (two symbols) is the simplest base system, because
with fewer than two symbols, we cannot represent change (and therefore
no information).

### Unsigned binary

When we talk about decimal, we deal with 10 digits---0 through 9
(that\'s where *deci*mal comes from). In binary we only have two digits,
that\'s why it\'s *bi*nary. The digits in binary are 0 and 1. You will
never see any 2\'s or 3\'s, etc. If you do, something is wrong. A bit
will always be a 0 or 1.

Counting in binary proceeds as follows:

`    0     (decimal 0) `\
`    1     (decimal 1) `\
`   10     (decimal 2) `\
`   11     (decimal 3) `\
`  100     (decimal 4) `\
`  101     (decimal 5) `\
`  ...`

An old joke runs, \"There are 10 types of people in the world. Those who
understand binary and those who don\'t.\"

The next thing to think about is what values are possible in one byte.
Let\'s write out the powers of two in a byte:

`2^7  2^6  2^5  2^4  2^3  2^2  2^1  2^0`\
`128  64   32   16   8    4    2    1 `

As an example, the binary number 10011001 is
$(1\times128) + (0\times 64) + (0\times 32) + (1\times 16) + (1\times 8) + (0\times 4) + (0\times 2) + (1\times 1) = 153.$
Note each of the 8 bits can either be a 0 or a 1. So there are two
possibilities for the leftmost bit, two for the next bit, two for the
bit after that, and so on: two choices for each of the 8 bits.
Multiplying these possibilities together gives $2^8$ or 256
possibilities. In *unsigned binary* these possibilities represent the
integers between 0 (all bits 0) to 255 (all bits 1).

All base systems work in the same way: the rightmost digit represents
the quantity of the base raised to the zeroth power (recall that
anything raised to the 0th power results in 1), and each digit to the
left represents a quantity that is base times larger than the one
represented by the digit immediately to the right. The binary number
1001 represents the quantity 9 in decimal, because the rightmost 1
represents $2^0=1$, the zeroes contribute nothing at the $2^1$ and $2^2$
positions, and finally the leftmost one represents $2^3=8$. When we use
different base systems it is necessary to indicate the base as the
subscript to avoid confusion. For example, we write $1001_2$ to indicate
the number 1001 in binary (which represents the quantity 9 in decimal).
The subscript 2 means \"binary\": it tells the reader that it does *not*
represent a thousand and one in decimal. This example also shows us that
representations have no intrinsic meaning. The same pattern of symbols,
e.g. 1001, can represent different quantities depending on the way it is
interpreted. There are many other ways to represent the quantity
$9_{10}$ (remember: read this as \"nine in base 10 / decimal\"); for
instance, the symbol 九 represents the same quantity in Chinese.

As the same quantity can be represented differently, we can often change
the representation without changing the quantity it represents. As shown
before, the binary representation $1001_2$ is equivalent to the decimal
representation $9_{10}$ - representing exactly the same quantity. In
studying computing we often need to convert between decimal
representation, which we are most familiar with, and binary
representation, which is used internally by computers.

### Binary to decimal conversion

Converting the binary representation of a non-negative integer to its
decimal representation is a straight-forward process: summing up the
quantities each binary digit represents yields the result.

$1001_2=1\times2^3+0\times2^2+0\times2^1+1\times2^0=8+0+0+1=9_{10}$

### Decimal to binary conversion

One task you will need to do in this book, and which computer scientists
often need to do, is to convert a decimal number to or from a binary
number. The last subsection showed how to convert binary to decimal:
take each power of 2 whose corresponding bit is a 1, and add those
powers together.

Suppose we want to do a decimal to binary conversion. As an example,
let\'s convert the decimal value 75 to binary. Here\'s one technique
that relies on successive division by 2:

`75/2   quotient=37   remainder=1`\
`37/2   quotient=18   remainder=1`\
`18/2   quotient=9    remainder=0`\
`9/2    quotient=4    remainder=1`\
`4/2    quotient=2    remainder=0`\
`2/2    quotient=1    remainder=0`\
`1/2    quotient=0    remainder=1`

We then take the remainders bottom-to-top to get 1001011. Since we
usually work with group of 8 bits, if it doesn\'t fill all eight bits,
we add zeroes at the front until it does. So we end up with 01001011.

## Binary mathematics

### Addition of binary numbers

In addition to storing data, computers also need to do operations such
as addition of data. How do we add numbers in binary representation?

Addition of bits has four simple rules, shown here as four vertical
columns:

`  0      0      1      1`\
`+ 0    + 1    + 0    + 1`\
`=========================`\
`  0      1      1      10`

Now if we have a binary number consisting of multiple bits we use these
four rules, plus \"carrying\". Here\'s an example:

`  00110101`\
`+ 10101100`\
`==========`\
`  11100001`

Here\'s the same example, but with the carried bits listed explicitly,
i.e., a 0 if there is no carry, and a 1 if there is. When 1+1=10, the 0
is kept in that column\'s solution and the 1 is carried over to be added
to the next column left.

`  0111100`\
`  00110101`\
`+ 10101100`\
`==========`\
`  11100001`

We can check binary operations by converting each number to decimal:
with both binary and decimal we\'re doing the same operations on the
same numbers, but with different representations. If the representations
and operations are correct the results should be consistent. Let\'s look
one more time at the example addition problem we just solved above.
Converting $00110101_2$ to decimal produces $53_{10}$ (do the conversion
on your own to verify its accuracy), and converting $10101100_2$ gives
$172_{10}$. Adding these yields $225_{10}$, which, when converted back
to binary is indeed $11100001_2$.

But binary addition doesn\'t always work *quite* right:

`  01110100`\
`+ 10011111`\
`==========`\
` 100010011  `

Note there are 9 bits in the result, but there should only be 8 in a
byte. Here is the sum in decimal:

`  116`\
`+ 159 `\
`=====`\
`  275`

Note 275 which is greater than 255, the maximum we can hold in an 8-bit
number. This results in a condition called **overflow**. Overflow is not
an issue if the computer can go to a 9-bit binary number; however, if
the computer only has 8 bits set aside for the result, overflow means
that a program might not run correctly or at all.

### Subtraction of binary numbers

Once again, let\'s start by looking at single bits:

`  0      0      1      1`\
`- 0    - 1    - 0    - 1`\
`========================`\
`  0     -1      1      0`

Notice that in the `-1` case, what we often want to do is get a 1 result
and borrow. So let\'s apply this to an 8-bit problem:

`  10011101`\
`- 00100010`\
`==========`\
`  01111011`

which is the same as (in base 10),

`  157`\
`-  34`\
`======`\
`  123`

Here\'s the binary subtraction again with the borrowing shown:

`  1100010`\
`  10011101`\
`- 00100010`\
`==========`\
`  01111011`

Most people find binary subtraction significantly harder than binary
addition.

## Other representations related to binary

You might have had questions about the binary representation in the last
section. For example, what about negative numbers? What about numbers
with a fractional part? Aren\'t all those 0\'s and 1\'s difficult for
humans to work with? These are good questions. In this and a couple of
other sections we\'ll look at a few other representations that are used
in computer science and are related to binary.

### Hexadecimal

Computers are good at binary. Humans aren\'t. Binary is hard for humans
to write, hard to read, and hard to understand. But what if we want a
number system that is easier to read but still is closely tied to binary
in some way, to preserve some of the advantages of binary?

One possibility is *hexadecimal*, i.e., base 16. But using a base
greater than 10 immediately presents a problem. Specifically, we run out
of digits after 0 to 9 --- we can\'t use 10, 11, or greater because
those have multiple digits within them. So instead we use letters: A is
10, B is 11, C is 12, D is 13, E is 14, and F is 15. So the digits
we\'re using are 0 through F instead of 0 through 9 in decimal, or
instead of 0 and 1 in binary.

We also have to reexamine the value of each place. In hexadecimal, each
place represents a power of 16. A two-digit hexadecimal number has a
16\'s place and a 1\'s place. For example, D8 has D in the 16\'s place,
and 8 in the 1\'s place:

`16^1  16^0  <- hexadecimal places showing powers of 16`\
`16    1     <- value of these places in decimal (base 10)`\
`D     8     <- our sample hexadecimal number`

So the hexadecimal number D8 equals
$(13 \times 16) + (8 \times 1) = 216$ in decimal. Note any two digit
hexadecimal number, however, can represent the same amount of
information as one byte of binary. (That\'s because the largest
two-digit hex number
$FF_{16} = (15 \times 16) + (15 \times 1) = 255_10 = 11111111_2$, the
same maximum as 8 bits of binary.) So it\'s easier for us to read or
write.

When working with a number, there are times when which representation is
being used isn\'t clear. For example, does 10 represent the number ten
(so the representation is decimal), the number two (the representation
is binary), the number sixteen (hexadecimal), or some other number?
Often, the representation is clear from the context. However, when it
isn\'t, we use a subscript to clarify which representation is being
used, for example $10_{10}$ for decimal, versus $10_2$ for binary,
versus $10_{16}$ for hexadecimal.

Hexadecimal numbers can have more hexadecimal digits than the two we\'ve
already seen. For example, consider $FF0581A4_{16}$, which uses the
following powers of 16:

`16^7  16^6  16^5  16^4  16^3  16^2  16^1  16^0`\
`F     F     0     5     8     1     A     4`

So in decimal this is:
$(15 \times 16^7) + (15 \times 16^6) + (0 \times 16^5) + (5 \times 16^4)$
$+ (8 \times 16^3) + (1 \times 16^2) + (10 \times 16^1) + (4 \times 16^0)$
$= 4,278,550,948$

Hexadecimal doesn\'t appear often, but it is used in some places, for
example sometimes to represent memory addresses (you\'ll see this in a
future chapter) or colors. Why is it useful in such cases? Consider a
24-bit RGB color with 8 bits each for red, green, and blue. Since 8 bits
requires 2 hexadecimal digits, a 24-bit color needs 6 hexadecimal
digits, rather than 24 bits. For example, `FF0088` indicates a 24-bit
color with a full red component, no green, and a mid-level blue.

Now there are additional types of conversion problems:

`* Decimal to hexadecimal`\
`* Hexadecimal to decimal`\
`* Binary to hexadecimal`\
`* Hexadecimal to binary`

Here are a couple examples involving the last two of these.

Let\'s convert the binary number 00111100 to hexadecimal. To do this,
break it into two 4-bit parts: 0011 and 1100. Now convert each part to
decimal and get 3 and 12. The 3 is a hexadecimal digit, but 12 isn\'t.
Instead recall that C is the hexadecimal representation for 12. So the
hexadecimal representation for 00111100 is 3C.

Rather than going from binary to decimal (for each 4-bit segment) and
then to hexadecimal digits, you could go from binary to hexadecimal
directly.

Hexadecimal digits and their decimal and binary equivalents: first, base
16 (hexadecimal), then base 10 (decimal), then base 2 (binary).

`16 10  2 <- bases`\
`===========`\
`0  0   0000`\
`1  1   0001`\
`2  2   0010`\
`3  3   0011`\
`4  4   0100`\
`5  5   0101`\
`6  6   0110`\
`7  7   0111`\
`8  8   1000`\
`9  9   1001`\
`A  10  1010`\
`B  11  1011`\
`C  12  1100`\
`D  13  1101`\
`E  14  1110`\
`F  15  1111`

Now let\'s convert the hexadecimal number D6 to binary. D is the
hexadecimal representation for $13_{10}$, which is 1101 in binary. 6 in
binary is 0110. Put these two parts together to get 11010110. Again we
could skip the intermediate conversions by using the hexadecimal and
binary columns above.

## Text representation

A piece of text can be viewed as a stream of symbols can be
represented/encoded as a sequence of bits resulting in a stream of bits
for the text. Two common encoding schemes are
ASCII code and
Unicode. ASCII code use one
byte (8 bits) to represent each symbol and can represent up to 256
($2^8=256$) different symbols, which includes the English alphabet (in
both lower and upper cases) and other commonly used symbols. Unicode
extends ASCII code to represent a much larger number of symbols using
multiple bytes. Unicode can represent any symbol from any written
language and much more.

## Image, audio, and video files

Images, audio, and video are other types of data. How computers
represent these types of data is fascinating but complex. For example,
there are perceptual issues (e.g., what types of sounds can humans hear,
and how does that affect how many numbers we need to store to reliably
represent music?), size issues (as we\'ll see below, these types of data
can result in large file sizes), standards issues (e.g., you might have
heard of JPEG or GIF image formats), and other issues.

We won\'t be able to cover image, audio, and video representation in
depth: the details are too complicated, and can get very sophisticated.
For example, JPEG images can rely on an advanced mathematical technique
called the discrete cosine transform. However, it is worth examining a
few key high-level points about image, audio, and video files:

1.  Computers can represent not only basic numeric and text data, but
    also data such as music, images, and video.
2.  They do this by digitizing the data. At the lowest level the data is
    still represented in terms of bits, but there are higher-level
    representational constructs as well.
3.  There are numerous ways to encode such data, and so standard
    encoding techniques are useful.
4.  Audio, image, and video files can be large, which presents
    challenges in terms of storing, processing and transmitting these
    files. For this reason most encoding techniques use some
    sophisticated types of compression.

### Images

A perceived image is the result of light beams physically coming into
our eyes and triggering nerves to send signals to our brain. In
computing, an image is simulated by a grid of dots (called *pixels*, for
\"picture element\"), each of which has a particular color. This works
because our eyes cannot tell the difference between the original image
and the dot-based image if the resolution (number of dots used) is high
enough. In fact, the computer screen itself uses such a grid of pixels
to display images and text.

\"The largest and most detailed photograph of our galaxy ever taken has
been unveiled. The gigantic nine-gigapixel image captures more than 84
million stars at the core of the Milky Way. It was created with data
gathered by the Visible and Infrared Survey Telescope for Astronomy
(VISTA) at the European Southern Observatory\'s Paranal Observatory in
Chile. If it was printed with the resolution of a newspaper it would
stretch 30 feet long and 23 feet tall, the team behind it said, and has
a resolution of 108,200 by 81,500 pixels.\"[^5]

While this galaxy image is obviously an extreme example, it illustrates
that images (even much smaller images) can take significant computer
space. Here is a more mundane example. Suppose you have an image that is
1500 pixels wide, and 1000 pixels high. Each pixel is stored as a 24-bit
color. How many bytes does it take to store this image?

This problem describes a straightforward but naive way to store the
image: for each row, for each column, store the 24-bit color at that
location. The answer is $1500\times1000$ pixels multiplied by 24
bits/pixel multiplied by 8 bits per 1 byte = 4.5 million bytes, or about
4.5MB.

Note the file size. If you store a number of photographs or other images
you know that images, and especially collections of images, can take up
considerable storage space. You might also know that most images do not
take 4.5MB. And you have probably heard of some image storage formats
such as JPEG or GIF.

Why are most image sizes tens or hundreds of kilobytes rather than
megabytes? Most images are stored not in a direct format, but using some
compression technique. For example, suppose you have a night image where
the entire top half of the image is black ((0,0,0) in RGB). Rather than
storing (0,0,0) as many times as there are pixels in the upper half of
the image, it is more efficient to use some \"shorthand.\" For example,
rather than having a file that has thousands of 0\'s in it, you could
have (0,0,0) plus a number indicating how many pixels starting the image
(if you read from line by line from top to bottom) have color (0,0,0).

This leads to a compressed image: an image that contains all, or most,
of the information in the original image, but in a more efficient
representation. For example, if an original image would have taken 4MB,
but the more efficient version takes 400KB, then the compression ratio
is 4MB to 400KB, or about 10 to 1.

Complicated compression standards, such as JPEG, use a variety of
techniques to compress images. The techniques can be quite
sophisticated.

How much can an image be compressed? It depends on a number of factors.
For many images, a compression ratio of, say, 10:1 is possible, but this
depends on the image and on its use. For example, one factor is how
complicated an image is. An uncomplicated image (say, as an extreme
example, if every pixel is black[^6]), can be compressed a very large
amount. Richer, more complicated images can be compressed less. However,
even complicated images can usually be compressed at least somewhat.

Another consideration is how faithful the compressed image is to the
original. For example, many users will trade some small discrepancies
between the original image and the compressed image for a smaller file
size, as long as those discrepancies are not easily noticeable. A
compression scheme that doesn\'t lose any image information is called a
*lossless* scheme. One that does is called *lossy*. Lossy compression
will give better compression than lossless, but with some loss of
fidelity.[^7]

In addition, the encoding of an image includes other metadata, such as
the size of the image, the encoding standard, and the date and time when
it was created.

### Video

It is not hard to imagine that videos can be encoded as series of image
frames with synchronized audio tracks also encoded using bits.

Suppose you have a 10 minute video, 256 x 256 pixels, 24 bits per pixel,
and 30 frames of the video per second. You use an encoding that stores
all bits for each pixel for each frame in the video. What is the total
file size? And suppose you have a 500 kilobit per second download
connection; how long will it take to download the file?

This problem highlights some of the challenges of video files. Note the
answer to the file size question is (256x256) pixels $\times$ 24
bits/pixel $\times$ 10 minutes $\times$ 60 seconds/minute $\times$ 30
frames per second = approximately 28 Gb (Gb means giga*bits*). This is
about 28/8 = 3.5 giga*bytes*. With a 500 kilobit per second download
rate, this will take 28Gb/500 Kbps, or about 56,000 seconds. This is
over 15 hours, longer than many people would like to wait. And the time
will only increase if the number of pixels per frame is larger (e.g., in
a full screen display) or the video length is longer, or the download
speed is slower.

So video file size can be an issue. However, it does not take 15 hours
to download a ten minute video; as with image files, there are ways to
decrease the file size and transmission time. For example, standards
such as MPEG make use not only of image compression techniques to
decrease the storage size of a single frame, but also take advantage of
the fact that a scene in one frame is usually quite similar to the scene
in the next frame. There\'s a wealth of information online about various
compression techniques and standards, storage media, etc.[^8]

### Audio

It might seem, at first, that audio files shouldn\'t take anywhere as
much space as video. However, if you think about how complicated audio
such as music can be, you probably won\'t be surprised that audio files
can also be large.

Sound is essentially vibrations, or collections of sound waves
travelling through the air. Humans can hear sound waves that have
frequencies of between 20 and 20,000 cycles per second.[^9] To avoid
certain undesirable artifacts, audio files need to use a sample rate of
twice the highest frequency. So, for example, for a CD music is usually
sampled 44,100 Hz, or 44,100 times per second.[^10] And if you want a
stereo effect, you need to sample on two channels. For each sample you
want to store the amplitude using enough bits to give a faithful
representation. CDs usually use 16 bits per sample. So a minute of music
takes 44,100 samples $\times$ 16 bits/samples $\times$ 2 channels
$\times$ 60 seconds/minute $\times$ 8 bits/1 byte = about 10.5MB per
minute. This means a 4 minute song will take about 40MB, and an hour of
music will take about 630 MB, which is (very) roughly the amount of
memory a typical CD will hold.[^11]

Note, however, that if you want to download a 40 MB song over a 1Mbps
connection, it will take 40MB/1Mbps, which comes to about 320 seconds.
This is not a long time, but it would be desirable if it could be
shorter. So, not surprisingly, there are compression schemes that reduce
this considerably. For example, there is an MPEG audio compression
standard that will compress 4 minutes songs to about 4MB, a considerable
reduction.[^12]

## Sizes and limits of representations

In the last section we saw that a page of text could take a few thousand
bytes to store. Images files might take tens of thousands, hundreds of
thousands, or even more bytes. Music files can take millions of bytes.
Movie files can take billions. There are databases that consist of
trillions or quadrillions of bytes of data.

Computer science has special terminology and notation for large numbers
of bytes. Here is a table of memory amounts, their powers of two, and
approximate American English word.

`1 kilobyte (KB) — `$2^{10}$` bytes — thousand bytes`\
`1 megabyte (MB) — `$2^{20}$` bytes — million bytes`\
`1 gigabyte (GB) — `$2^{30}$` bytes — billion bytes`\
`1 terabyte (TB) — `$2^{40}$` bytes — trillion bytes`\
`1 petabyte (PB) — `$2^{50}$` bytes — quadrillion bytes`\
`1 exabyte (EB) — `$2^{60}$` bytes — quintillion bytes`

There are still higher numbers or smaller quantities of these
types.[^13]

Kilobytes, megabytes, and the other sizes are important enough for
discussing file sizes, computer memory sizes, and so on, that you should
know both the terminology and the abbreviations. One caution: file sizes
are usually given in terms of *bytes* (or kilobytes, megabytes, etc.).
However, some quantities in computer science are usually given in terms
involving bits. For example, download speeds are often given in terms of
bits per second. \"Mbps\" is an abbreviation for mega*bits* (not
megabytes) per second. Notice the \'b\' in Mbps is a lower case, while
the \'b\' in MB (megabytes) is capitalized.

In the context of computer memory, the usual definition of kilobytes,
megabytes, etc. is a power of two. For example, a kilobyte is
$2^{10} = 1024$ bytes, not a thousand. In some other situations,
however, a kilobyte is defined to be exactly a thousand bytes. This can
obviously be confusing. For the purposes of this book, the difference
will usually not matter. That is, in most problems we do, an
approximation will be close enough. So, for example, if we do a
calculation and find a file takes 6,536 bytes, then you can say this is
approximately 6.5 KB, unless the problem statement says otherwise.[^14]

All representations are limited in multiple ways. First, the number of
different things we can represent is limited because the number
combinations of symbols we can use is always limited by the physical
space available. For instance, if you were to represent a decimal number
by writing it down on a piece of paper, the size of the paper and the
size of the font limit how many digits you can put down. Similarly in a
computer the number of bits can be stored physically is also limited.
With three binary digits we can generate $2^3=8$ different
representations/patterns, namely
$000_2, 001_2, 010_2, 011_2, 100_2, 101_2, 110_2, 111_2$, which
conventionally represent 0 through 7 respectively. Keep in mind
representations do not have intrinsic meanings. So three bits can
possibly represent seven different things. With n bits we can represent
$2^n$ different things because each bit can be either one or zero and
$2^n$ are the total combinations we can get, which limits the amount of
information we can represent.

Another type of limit is due to the nature of the representations. For
example, one third can never be represented precisely by a decimal
format with a fractional part because there will be an infinite number
of threes after the decimal point. Similarly, one third can not be
represented precisely in binary format either. In other words, it is
impossible to represent one third as the sum of a finite list of power
of twos. However, in a base-three numbering system one third can be
represented precisely as: $0.1_3$ because the one after the point
represent a power of three: $3^{-1}$.

## Notes and references

```{=html}
<references />
```

[^1]: Analog at Wiktionary.

[^2]: Actually, it\'s more complicated than that because some devices,
    including some digital radios, intermix digital and analog. For
    example, a digital radio broadcast might start in digital form,
    i.e., as a stream of numbers, then be converted into and transmitted
    as radio waves, then received and converted back into digital form.
    Technically speaking the signal was *modulated* and *demodulated*.
    If you have a *modem* (*mod*ulator-*dem*odulator) on your computer,
    it fulfills a similar function.

[^3]: Actually we need not only data, but a way to represent the
    algorithms within the computer as well. How computers store
    algorithm instructions is discussed in another chapter.

[^4]: Of course how a 0 or 1 is represented varies according to the
    device. For example, in a computer the common way to differentiate a
    0 from a 1 is by electrical properties, such as using different
    voltage levels. In a fiber optic cable, the presence or absence of a
    light pulse can differentiate 0\'s from 1\'s. Optical storage
    devices can differentiate 0\'s and 1\'s by the presence or absence
    of small \"dents\" that affect the reflectivity of locations on the
    disk surface.

[^5]: 1

[^6]: You might have seen modern art paintings where the entire work is
    a single color.

[^7]: See, for example,
    2 for examples
    of the interplay between compression rate and image fidelity.

[^8]: For example, see
    3 and the
    links there.

[^9]: This is just a rough estimate since there is much individual
    variation as well as other factors that affect this range.

[^10]: Hz, or *Hertz*, is a measurement of frequency. It appears in a
    variety of places in computer science, computer engineering, and
    related fields such as electrical engineering. For example, a
    computer monitor might have a refresh rate of 60Hz, meaning it is
    redrawn 60 times per second. It is also used in many other fields.
    As an example, in most modern day concert music, A above middle C is
    taken to be 440 Hz.

[^11]: See, for example, 4 for
    more information about how CDs work. In general, there is a wealth
    of web sites about audio files, formats, storage media, etc.

[^12]: Remember there is also an MPEG video compression standard. MPEG
    actually has a collection of standards: see Moving Picture Experts
    Group on
    Wikipedia.

[^13]: See, for example, binary
    prefixes.

[^14]: The difference between \"round\" numbers, such as a million, and
    powers of 10 is not as pronounced for smaller numbers of bytes as it
    is for larger. A kilobyte is $2^{10}=1024$ bytes, which is only 2.4%
    more than a thousand. A megabyte is $2^{20} = 1,048,576$ bytes,
    about 4.9% more than one million. A gigabyte is about 7.4% bytes
    more than a billion, and a terabyte is about 10.0% more bytes than a
    trillion. In most of the file size problems we do, we\'ll be
    interested in the approximate size, and being off by 2% or 5% or 10%
    won\'t matter. But of course there are real-world applications where
    it does matter, so when doing file size problems keep in mind we are
    doing approximations, not exact calculations.


# Foundations of Computer Science/Algorithms and Programs

## Algorithms and Programs

An algorithm can be defined
as a set of steps used to solve a specific problem. For example, a cook
may use a recipe when preparing a specific type of food. Similarly, in
computer science, algorithms are the conceptual solutions used to create
programs. It is important to distinguish an algorithm from a program.
The implementation of an algorithm is known as a
program.

### Defining information processes

Computer is about information processes. Once information is represented
concretely using different patterns of symbols it can be processed to
derive new information. We learned that computers use the binary system
internally to represent everything as sequence of bits - zeros and ones.
Chapter 1 of the Blown to Bits
book
talks about the digital explosion of bits as the result of the
innovations in computing and technologies, which enable us to turn
information into bits and shared them with unprecedented speed.

Creating information processes is the topic of this chapter. We will
learn that information processes start with conceptual solutions to
problems and then can be implemented (coded) in ways understandable to
machines. The conceptual solutions are called algorithms and the
executable implementations are called programs.

### What is an algorithm?

Algorithm is a rather fancy
name for a simple idea: a step-by-step solution to a problem. Avi
Wigderson once said algorithm is a common language for nature, human,
and computer. The idea has been around for a long time. You are already
familiar with many algorithms, such as tying your shoes, making coffee,
send an email, and cooking a dish according to a recipe. Algorithms in
computing are designed for computers to follow. Imagine we have built a
machine that can perform the single digit addition procedure described
in chapter one. Recall the procedure performs the addition using simple
table lookup. If we give the machine two digits and ask it to perform
the operation, it gives two digits back as the answer. Of course the
numbers in the inputs and the output have to be represented (encoded)
properly. Even though the machine doesn\'t understand addition, it
should be able to perform the addition correctly. However, the machine
will not perform the addition unless it is instructed to do so. The
command (with input values) that signals the machine to perform an
addition is called an instruction. We have imagined that it is not hard
to use the addition procedure to create other more complex procedures
that can perform more impressive activities. Before we can create such
procedures we must identify a problem and find a conceptual solution to
it. Often time the conceptual solution is one that can be carried out
manually by a person. This conceptual solution is an algorithm.

### Why study algorithms?

Algorithm is a center piece in the computer science discipline. As we
discussed in chapter one, computing can be done blindly or purely
mechanically by a simple device. The intelligence of any computation
(information process) lies in the algorithm that defines it.

For an algorithm to be useful, it must be correct - the steps must be
logical and specific for machines to carry out - and efficient - it must
finish in a reasonable amount of time. The correctness and efficiency of
algorithms are two key issues in the study of algorithms.

### Programs are implemented algorithms

Studying algorithms allows us to solve problems conceptually regardless
of the machines that carry out the solutions. An algorithm must
communicate a conceptual solution in a unambiguous and human
understandable fashion. A notational system for describing algorithms
should allow us to describe and reason with ideas on paper. Once an
algorithm\'s correctness is verified on paper, it can be implemented as
a program understandable to a particular machine.

### Formal definition of algorithm

Alan Turing is the first
person who studied algorithms mathematically by creating a universal
machine model, later called Turing machine. He also proved that
computation is unavoidable in circumstances - we will have to perform
the computation to the result, which separates computing from
mathematics (the birth of computer science). The turing machine can
represent/encode information in a standard form and interpret and
updates the representation according to rules (algorithms), which is
part of the representation. This machine model is simple yet powerful.
In fact, it is the most powerful computing model known to computer
scientist. The turing machine can perform any computation done by any
machine we could ever build. Turing machine
equivalents
is defined based on this idea.

The turing machine model allows us to study algorithms in abstraction.
For instance, we can view each algorithm as a state
machine: an
algorithm always starts with a state - represented by a data
representation of the input and the internal information - and move
through a number of states to its final state as the result of
performing operations prescribed in the algorithm. When the number of
possible initial states approach infinity the same algorithm can
generate potentially infinite number of computation. This explains why
it is hard to verify the correctness of an algorithm through testing as
the initial states can be too many to exhaustively enumerate.

### Define algorithms

An algorithm is simply a set of steps that allow us to solve a
particular problem. A step is a unit of work that is unambiguous and can
be done in a fixed amount of time. For example bring a pot of water to
boil is a step in the tea-making process/algorithm. In computing we deal
with representations of information (data), so a step can be adding two
integers and storing the result to a variable. We will explain how to
define and use variables later.

The definition of a unit of work depends on what the agent, who performs
work, can do. Algorithms in computing are necessarily informed by the
capability of the computing machines. Recall that algorithms must be
implemented/described in a programming language understandable to a
machine before the machine can perform the task. There are many
different programming languages, therefore different ways to express the
same algorithm. The only language understandable to a specific machine
is called the machine language. Machine languages are written in
instructions consisting of zeros and ones (binary bits) because
computers are fundamentally machines that can manipulate two types of
symbols. Each different type of machine is designed to understand its
own native language---patterns of zeros and ones---because they can have
very different computing
hardware.
As you can imagine writing programs in machine languages can be very
hard. Normally we write programs to express our algorithms in high level
languages - languages that are close to our natural language, e.g.
English. Then we use tools (compilers and interpreters) to translate our
programs in higher level languages to machine languages, which is
analogous to using a personal interpreter when we travel abroad and
don\'t understand the native language. To run the same program on a
different machine we can simply recompile it or use a different
interpreter. High level languages hides the differences between machines
to allow us to write programs in a machine independent way, which is a
huge time saver. When we write programs in high level languages we use
an abstraction that is supported by all computers. For instance if a
high level language allows the expression of an addition we assume it
can be done by all computers.

Programming languages (high level or machine level) are tools for
expressing algorithms to machines. When we create algorithms to solve
problems conceptually we want to create them independent of the
languages. A well-designed recipe ought to work for different cooks in
different kitchens. So the steps or units of work must be defined in
terms of a higher abstraction - a set of common data structure,
operations and control structures that all languages support. By
creating algorithms conceptually using abstractions allows us humans to
think on a higher level closer to the problem domain we know. When an
algorithm is implemented in a particular language the abstract steps can
be mapped to the specific expression in the language. We are confident
the chain of tools we have can translate the solution to the machine
level executable code. The following diagram shows that the same
algorithm can be implemented in a variety of programming languages and
the result programs can be executed on different machine models (maybe
after some translation).

!This diagram shows that the same algorithm can be implemented in a
variety of programming language and then result programs can be run on
different machine
models.

Here are the common operations and control structures we can assume all
high level languages support:

-   data structures: variables of single value and list of values.
-   operations: arithmetic operations, comparisons, and relational
    operations (and, or, and not)
-   control structures: sequential (one after another), conditional
    (selective on a condition), and repetition

Here is an algorithm defined in pseudo
code (natural language). This
algorithm finds the largest number in a list of numbers with the
following steps:

1.  set max (a variable) to the value of the first number in the list
    (store and retrieve values).
2.  go through the list one number at a time comparing each number to
    max, if the number is greater than max replace max with the number
    (conditional and repetition).
3.  the value stored in max is the answer.

We know the solution is correct because it is so simple. We know how to
carry it out manually, but we would certainly not solve the problem
using the process expressed in psuedo code. It is necessary to design
and express the algorithm in this detailed fashion for computers. Keep
in mind that computers are machines that perform computation
mechanically and therefore the instructions must be specific. The psuedo
code, even though in natural language, must use the aforementioned
constructs (data structures, operations, and control structures) that
are common to computers. You may wonder why we can not ask the computer
to look at the whole list and identify the largest number as we humans
do. Computers are simple machines that can not think. They are designed
to perform simple operations, e.g. adding two digits via symbol
manipulation. Even we, as human beings, will not be able to scan a long
list, say a million numbers, and find the largest number at a glance.
The algorithm we write must be expressed in terms of what a computer can
do and are scalable to inputs (data sets) of arbitrary sizes. The
algorithm we just studied can deal with a list of any size. In fact, it
makes little difference to a computer whether the list has three numbers
or three million numbers.

Another way to express the same algorithm is to use a graphical notation
called flow-chart. !This flow-chart shows the steps involved in finding
the largest number in a list of
numbers.
This chart shows the logic of the solution more clearly. There are two
conditionals - the checking of a condition and corresponding actions
taken as the result. The top-most conditional defines a repetition
(loop) because their an unconditional branching back to the conditional
as expressed in the arrow with no label.

Both the pseudo code and the flow chart describe the same solution to
the same problem. They are different representations of the same idea.
The following figure shows an implementation of the algorithm in
Scratch. !This stack of blocks scratch finds the largest number in a
list of
numbers.
As you can see, a concrete implementation of an algorithm must use the
building \"blocks\" available in the particular implementation language.
What is not shown in the figure is the part where the list is populated
with data from a file or user input. The structure of the code resembles
that of the flow-chart.

In summary constructing and studying algorithms allows us to study
(algorithmic) solution to problems in a language and computing
environment neutral way. We could use the \"finding largest number\"
algorithm as a single block to form more complex algorithms, but we need
to keep in mind this block is not a unit of work as the number of steps
involved depends on the input size (the length of the list). We will
revisit this topic in the future when we talk about functional
decomposition (to keep algorithms simple) and algorithm complexity
analysis (count the cost of algorithms).

Programs Each software program boils down to two components - data
(structure) and algorithm. We will study some fundamental algorithms and
data structures in computer science.

### Example algorithms

#### Image enconding/representation

Follow this
<http://csunplugged.org/sites/default/files/activity_pdfs_full/unplugged-02-image_representation.pdf>
image representation
activity to see how images
are encoded, transmitted, and reproduced in fax machines.

#### Error detection

Follow this
<http://csunplugged.org/sites/default/files/activity_pdfs_full/unplugged-04-error_detection.pdf>
error detection activity to see
the algorithm works to detect and also correct single bit errors.

A similar algorithm, Luhn
algorithm, is used to
validate credit card numbers.

#### Text compression

Text compression is another
important task in computing. The following activity demonstrates how a
compression algorithm works:
<http://csunplugged.org/sites/default/files/activity_pdfs_full/unplugged-03-text_compression.pdf>

#### Searching

Why is searching important? We do it on a daily basis. It is good
business too. Google\'s mission is to organize the world\'s information
and make it universally accessible and useful. Obviously to be able to
find the information we need fast is very useful and profitable.

We can always find a piece of information by going through a list of
them sequentially checking each one of them. Could you describe the
algorithm using either the pseudo code or the flow-chart notation?
Structure-wise this algorithm should resemble the \"find largest
number\" algorithm. This algorithm needs two inputs: the list and the
target item we are looking for. The repeated steps are fetching the next
item and comparing it to target. The piece of information used in the
comparison is also known as the key because it determines whether a
search is successful or not. For instance, if we have a list of students
we can search a student by last name, birthdate, or shoe size, which are
search keys.

A sequential search is straight-forward, but it can be costly if we need
to perform it very often. However, if the list is ordered by the search
key we can use a much better algorithm by taking advantage of this
ordered property of the data. Think about the index of a book, a phone
book, or a dictionary. They are all ordered somehow. For instance, home
phone numbers in a phone books are usually ordered by owner\'s last
names and business phone numbers are ordered by business types. Entries
in a dictionary or the index of a book are ordered alphabetically. How
does this orderedness help us when we search for information? It allows
us to guesstimate where the search target is located. The number
guessing
game
illustrate the idea well. If the list of numbers are random but ordered
increasingly we can always guess the target number is in the middle of
the search range (initially the whole list). If we are not lucky, we can
eliminate half of the candidates - if the number in the middle is
greater than the search target the new search range will be the first
half of the previous search range, otherwise the second half. Imagine
the reduction in the number of comparisons! We will study this algorithm
in much detail when we discuss algorithm complexities.

### Social impact

Please watch the following video How algorithms shape our
world. Could you explain
in what ways algorithms are shaping our world?

In the world of computing, we store and process data (representations of
information) by
quantifying information.
This quantification process reduces the world to what can be counted and
measured and emphasize abstraction and efficiency (speed). We must not
be fooled to believe the abstractions of reality are true reality as in
abstractionism. As
Frederick Brooks warns us "Models are intentional oversimplifications to
help us with real-life problems that are frighteningly complicated. The
map is not the terrain."


# Foundations of Computer Science/Algorithm Design

## Algorithm Design

Algorithm design is a
specific method to create a mathematical process in solving problems.
One powerful example of algorithm design can be seen when solving a
Rubik\'s cube. When solving a Rubik\'s cube (of any size), it is
important to know the step by step instructions or algorithm to reach an
efficient solution. This is where algorithm design comes into place.
There are designs that breakdown the seemingly complex solution by
addressing each layer (First, Middle, and Last and colors.

Please follow the link on solving the last layer of a 3X3 Rubik\'s
cube).

### Approaches to algorithm design

#### Top Down

The top down approach to design is starting by examining the full
problem or one way to think of it is to look at the big/whole picture
first. Once you have assessed the main problem then you divide the
problem into smaller components or parts.

The next portion of the top down approach is to begin testing.
Initially, we will have portions that are missing due to focusing on the
bigger picture. In situations where parts of the problem have not been
solved stubs or placeholders are used to as temporary holders.

One way to think about the top down approach is in a hierarchical
setting such as a general command his troops. The general will breakdown
a mission by assigning each soldier with a specific task to complete
that in turn contributes to a critical part of the overall mission.

#### Bottom Down

The bottom down approach to algorithm design starts with the smallest
units or parts of a problem. This approach solves the smallest units
first and then gradually builds out the next layer or solution. Using
this method ensures that the smallest unit has been successfully tested
and therefore, when you start solving or implementing the next
sub-solution that it will work due to the previous layers working
successfully.

One example is building a car. Each piece of the car is engineered,
created, and tested piece by piece. Knowing that the smaller parts work
correctly the parts are then gradually added on an assembly line. As the
parts are added, you know that the smaller components work due to
thoroughly testing each piece. Eventually, as you walk through this
process the end result is a properly functioning car.

### Algorithm Design: Building Blocks

There are basic logic structures and operations involved in algorithm
design. Building blocks are necessary to decide how we want to
manipulate units of work. The basis of every algorithm is steps or
blocks of operations. These steps/blocks of operation can be as simple
as adding two numbers together. However, these blocks of operation can
also be complex, for example, finding the maximum value in a list of
numbers.

Logic structures in important for organizing steps into a
process/solution. The following four basic logic structures describe the
type of blocks used for creating algorithms:

-   procedure/function call - one example would be a single block in
    Scratch
-   sequence - in order to create a sequence you need a stack of blocks
-   alternatives - use of the if-then-else blocks to indicate alternate
    solutions for a particular problem
-   iteration - using the \"repeat\", \"for\", and \"forever\" blocks to
    build loops to solve problems

Most languages have programming constructs for basic operations and
logic structures. In order to understand programming constructs you must
first learn about constructed
language. According
to Wikipedia, a constructed language \"is a language whose phonology,
grammar, and vocabulary has been consciously devised for human or
human-like communication, instead of having developed naturally\".
Similarly, a programming
construct is
\"designed to communicate instructions to a machine, particularly a
computer.\"


# Foundations of Computer Science/Algorithm Complexity

## Algorithm Complexity

\"An algorithm is an abstract recipe, prescribing a process that might
be carried out by a human, by computer, or by other means. It thus
represents a very general concept, with numerous applications.\"---David
Harel, \"Algorithmics - the spirit of computing\".

We have learned that algorithms are conceptual solutions to problems.
Computing algorithms can be described/defined in terms of the common
units of work that computers can do so that they are neutral to
programming languages and the execution environment (computers).
Algorithm writing is such a creative process as Francis Sullivan noted
that \"for me, great algorithms are the poetry of computing. Just like
verse, they can be terse, allusive, dense, and even mysterious. But once
unlocked, they cast a brilliant new light on some aspects of
computing.\" You have seen example algorithms documented using abstract
notations such as pseudo code and flowchart.

Once algorithms are implemented/coded, i.e. represented by concrete
symbols, they become alive in program that embody them. Programs are
executable/runnable by computers. The ability to run different programs
that implements all kinds algorithms is a unique feature of computers as
machines. Usually when we buy a machine, e.g. an appliance, we assume it
has a set of well defined functions it can perform. For example a
microwave oven is supposed to warm and cook our food and nothing more.
We don\'t expect a microwave oven ever to be able to wash the clothes
for us. A computing machine (a computer) is different. We expect it to
perform the function whichever program makes it to do - the
functionality of a computer is extensible. If we liken a computer to a
car, programmers are drivers who can make the car do different things
(to a certain extent) and users are like passengers taking advantage of
the things the car can do. This is another reason why everyone needs to
learn computer programming because it gives you the freedom to make the
computer do different thing.

### Correctness of Algorithms

Algorithms must be correct to be useful. We must examine our algorithms
carefully to remove errors before using them to create programs. If an
algorithm has errors, it is impossible to create correct programs. We
often remind students that if their algorithms do not work on paper they
won\'t work on a computer. So we must work out our algorithms on paper
first.

Even if the design of an algorithm is correct we may introduce errors
during the programming process. As any natural language a programming
language has its own syntax consisting of grammatical rules for using
the language. When we violate such rules we introduce syntax errors in
our program. This type of errors are easy to fix, in fact most modern
program editors can detect and warn us about such errors. Another type
of errors are logic errors, which result from the misuse of the
language. In other words, our grammatically correct program doesn\'t
make sense or make the wrong sense to the computer. For example, a
recipe can be unclear or misleading even though all sentences are
grammatically correct. You may think that computers can also make
mistakes when running programs that are logically. This is true but very
rarely this is the case especially with modern computers equipped with
error detection mechanisms. We generally assume computers don\'t make
mistakes. If the program doesn\'t generate the correct answer, it is the
result of human errors.

We also call logic errors software
bugs. The original \"bug\"
is, in fact, a hardware failure - a moth caught in a electromechanical
computer. Now bugs are generally used to refer to any error/failure in
computer systems, both in hardware and in software. When a computer
program is buggy, it will produce erroneous results or crash as the
computer may not know what to do next. Another more subtle bug may cause
the computer program to never finish, known as the infinite loop, which
is obviously not what we wanted.

Bugs are almost inevitable as humans make mistakes. Then, how do we fix
bugs to ensure the correctness of our programs? We must test our
programs to verify its correctness. A test consists of a sample input to
a program and the desired output from the program. We can run a test by
subject the program to the sample input and collect the actual output.
If the actual output matches the desired output the program passes the
test, otherwise their is a bug in the program or in the test (tests can
be buggy too). We usually use a set of tests (a test suite) to exercise
different parts of the algorithm. This process is called debugging as
expressed in the following pseudo code:

`for each test in the test suite`\
`  run and compare the actual output to the desired output`\
`  if they match move on to the next test`\
`  otherwise fix the bug and repeat the whole process`

Note that it is very difficult to make our tests exhaustive except for
very simple programs. When a program becomes larger and more complex the
number of tests need to cover all possible cases grows very fast. As
Dijkstra said
\"program testing can be used to show the presence of bugs, but never to
show their absence!\" There are techniques for proving the correctness
of programs. For instance, microcode for computer processors are often
proved correct via formal
verification.

### \"Goodness\" of algorithms

There are usually always multiple ways to solve the same problem and,
therefore, multiple algorithms to make computers solve the problem for
us. In addition to solving the problem correctly we want to be able to
compare/evaluate algorithms that solve the same problem. We must define
the criteria/measure for \"goodness\" in algorithm design. Such criteria
or measures can include simplicity, ease of implementation, speed of
execution, or preciseness of answers. In computing we care the most
about the solution speed because a fast algorithm can solve larger
problem or more problems in the same amount of time. This is also known
as efficiency - an economic measure of many processes. Often time he
usefulness of many programs depends on the timeliness of the results.
For instance, a program that takes 24 hours to predict the next day\'s
weather is almost useless.

Given an algorithm how do we measure its \"speed\"? One possible
approach is to implement the algorithm, run the result program, and
measure its execution time - elapsed time between the start and the end
of a program run. There are some challenges in this approach. First the
algorithm must be implemented, which can a serious undertaking.
Secondly, to run two programs to compare their execution time we must
subject them to the same input (a.k.a a workload, e.g a list of one
million numbers to be sorted) and the size of the input is never ideal.
Thirdly, the \"speed\" of a running program is influenced by the
execution environment, such as the machine\'s hardware configuration.
Take a recipe for example. Different cooks will surely spend different
time following it. The amount of food needed will surely magnify the
difference - as we increase the amount of ingredients the lengthy
recipes will take even longer to follow. But there are intrinsic
characteristics of the recipes that affect the preparation time. If a
recipe involves beating eggs can instruct the cook to break each egg and
scramble it individually or break all eggs first and scramble them
together. Obviously the first method is slower due to the additional
steps involved. Algorithms in computing exhibit similar characteristics.
Recall that algorithms must be defined in terms of the units of work
(steps) computers can perform so that it is straightforward to implement
them in programming languages. The way the steps are ordered and
organized in algorithms can significantly affect the execution time of
the programs implement the algorithms.

Since an algorithm is a conceptual solution to a problem, we want to
study their \"speed\" in an abstract sense without actually implementing
them. This is known as algorithm analysis in computer science. In this
approach we take an algorithm described in pseudo code or flow chart,
count the number of steps (units of work), which is always a function of
the input size. In the aforementioned example recipe the time it takes
to follow the first method (break each egg and scramble it individually)
is directly proportional to the number of eggs involved. In fact if only
one egg is needed, there is no difference between the two methods.
Instead of measuring the steps taken for a particular input size, we
focus on the relationship function between the number of steps and the
input size, which shows the pattern in which the amount of work (cost)
grows as the input size increases. Such functions are also known as
growth functions. Then, we apply asymptotic
analysis, which
compare functions as inputs approach infinity, to simplify the functions
because as the input size approaches infinity the difference between the
units of works disappears (we can assume breaking an egg and scrambling
it take the same amount of time) and the cost of most \"complex\" part
of the task will dominate (a part that repeats a step 10 time will dwarf
any step that is only done once) the total cost. For example, in a
recipe we have a one quick step (takes 0.0001 seconds per serving) that
repeats 10 times and a slow step (takes 10 seconds per serving) doesn\'t
repeat, an amount of serving of N (input size) would cost $0.0001*10*N$
total seconds on the repeated steps and would always cost 10 seconds on
the slow step. When N is bigger than 10000, the repeated part would cost
more than the slow part. In asymptotic analysis we can ignore the slow
step because its contribution to the total cost is negligible when N
approaches infinity. With simplify growth functions we can put them into
categories considering algorithms in each category to have similar
performance characteristics. This type of analysis is not completely
precise but it can be very useful in studying the nature of algorithms
and predicting their performances. We learn how to put algorithms into
categories and rank their performances according to the categories they
are in. We denote each category using big O notation.

In summary we have discussed the following few key concepts regarding
algorithm complexity analysis:

-   Algorithms are conceptual and abstract solutions to problems and
    programs are concrete executable code computers can run. We can not
    run algorithms on computers; we can only run programs that implement
    algorithms. Therefore, the performance of an algorithm is by
    definition abstract (can not be concretely defined or measured).
-   The goodness measure of an algorithm has to be an intrinsic
    characteristic of the algorithm itself - something that reflects the
    \"cleverness\" of the design regardless the implementation details
    and the future execution environments.
-   From an economic perspective we care about the cost of algorithms
    both in terms of time and space. We will focus only on the time cost
    in this course. We can not actually measure the time cost of
    algorithms, but we can study the relationship between the time cost
    and the input size, which reflects the internal complexity (or
    cleverness) of algorithms. We represent such relationships as growth
    functions with the input size as the variable and the total cost as
    the value. The growth functions can be simplified by keeping only
    the dominate term (asymptotic analysis) because other terms won\'t
    matter when the input becomes really large (eventually approaching
    infinity). Simplified growth functions are put into categories and
    algorithms can be rank by the categories they belong to.
-   Algorithms in the low complexity category will perform better than
    algorithms in the higher complexity categories when the input size
    is sufficiently large. We care about large input sizes because any
    algorithm can solve a small problem fast. With this algorithm
    analysis technique we can evaluate and compare algorithms to predict
    the relative performance of the programs implementing such
    algorithms before actually implementing any of them.
-   Efficiency, as another term often used, is reversely proportional to
    complexity. A more complex algorithm is less efficient because it
    makes less efficient uses of the computing resources by being more
    complex.

### Examples

There are at least two ways to calculate the sum of all numbers between
1 and N. The following are two algorithms:

-   Algorithm 1: add all the numbers up manually, one by one
-   Algorithm 2: calculate the result using this formula $(N+1)*(N/2)$

Consider the following question: if we carry out the two algorithms
manually, which one would run faster if

-   N = 2?
-   N = 100?
-   N = 1,000,000?

Lets see how a algorithms behave (after implemented) on a computer. The
following script implements algorithm 1 using a block that iterates
through all the numbers in the range and adding them to the sum one at a
time. !A Snap! script that adds all the numbers between two
numbers.
!A Snap! reporter block that adds all the numbers between two numbers
and reports the
sum.
The same problem can be solved using algorithm 2, which use a function
to calculate the result as shown in the following script and the report
block. !A Snap! reporter block that calculate the sum of all numbers in
range and reports the
result.
!A Snap! block that calculates the sum of all numbers in a
range.

Both scripts (programs) take the same input - two numbers that define a
range. The number of numbers in the range is the input size or more
properly the problem size. We assume the units of work are arithmetic
operations $+-*/$, assignment operations, and report operation. Such an
assumption is reasonable because in deed those operations takes the same
to to perform regardless the operands. If you run the programs and try
different input size you will observe that as you increase the input
size the execution time of the first program increases steadily whereas
the second program shows not change in execution time. Can we predict
such behaviors?

Lets apply algorithm analysis to these two algorithms. The first
algorithm loops through the numbers to get the sum, therefore the
relationship function between cost (the number of steps - additions) and
the input size is $f=a+b*N$ assuming N is the input size and a and b are
the cost for creating the script variable and the cost for an addition.
Note that both a and b are constant because they don\'t change for a
given computer. We can simply the function to $f=N$ because when N
approaches infinity the constants don\'t matter anymore. We assign
algorithms with this type of growth function into the leaner algorithm
to the leaner this type of functions linear time category denoted by
$O(N)$. The second algorithm always takes the same amount of time
because it performs the same set of operations regardless of the input
size. It belongs to the constant time category denoted by $O(1)$.

Lets consider another problem: are numbers in a list distinct? Here is a
straight forward algorithm in pseudo code:

-   Step 1: compare the first number with all of the other numbers in
    the list. If, at any point, the same number is seen, stop and answer
    NO.
-   Step 2: repeat Step 1 by taking the next number from the list and
    comparing it with all of the other numbers.
-   Step 3: after using all numbers in the list, stop and answer YES.

The input size is the size of the list. The more numbers in the list the
longer it takes to answer the question. According to algorithm it is
possible that the first comparison find two identical numbers, which
answer the questions right a way. This is good design because at that
point it is unnecessary to continue with the rest of the algorithm. When
we analyze algorithms we focus on worst cases because the performance of
an algorithm is dependent from the actual input and we should not rely
on luck! So the relationship (growth) function between the cost and the
input size for this algorithm should be $f=N*(N-1)$ because in the worst
case (all numbers are unique) the algorithm has to compare each number
to all the other numbers in the list before it arrive at the answer. To
simply the function by keeping only the largest dominating term we get
$f=N^2$, which puts the algorithm in the quadratic category $O(N^2)$. If
you are interested you can reproduce and run the following script (an
implementation of the algorithm) to verify the predicted performance
characteristics of the algorithm. !A Snap! reporter block that tests
whether all the numbers in a list are
unique.
!A Snap! script that tests whether all the numbers in a list are
unique.

Lets look at an algorithm from another category: print all n-digit
numbers?

`For each number in 0, 1, 2, 3, ..., 9 `\
` use the number for the first digit`\
` enumerate all n-1 digit numbers`\
` append each every n-1 digit number to the first digit to form a n digit number`

This example is a little contrived but it demonstrate a new performance
category and is very useful when we discuss encryption techniques. The
algorithm is complex simply because the amount of output (all n digit
numbers) it has to generate. So the cost of the output part will
dominate the total cost. To generate all n digit numbers we must first
generate all n-1 digit numbers. To generate all n-1 digit numbers we
must first generate all n-2 digit numbers and so on. It is easier to
study the process backward. Assuming outputting a number is the unit of
work (it takes the same time output a number) the cost to generate all
one digit numbers is 10. To generate all two digit numbers we have to
output $10*10=10^2$ numbers. For three digit numbers the cost is $10^3$.
Did you see a pattern? To generate all n digit numbers it costs $10^n$.
This type of algorithms goes much faster than the quadratic ones because
the input size is in the exponent. They belong to the exponential time
category denoted as $O(2^N)$. The root (in this case 2) doesn\'t matter.
Because all exponential functions are more similar to each other than
quadratic functions or linear functions.

Here is a YouTube video illustrating the performance difference between
the bubble sort algorithm and the quick sort algorithm:
<https://www.youtube.com/watch?v=aXXWXz5rF64>


# Foundations of Computer Science/Abstraction and Recursion

## Abstraction and Recursion

Programming is easy as long as the programs are small. Inevitably our
programs will grow larger and larger as we create them to solve
increasingly complex problem. One technique we use to keep our
algorithms and programs simple is
abstraction, which is an
idea widely used in many fields such as art, math and engineering. In
this chapter we will study how to apply this technique is algorithm
design and programming.

An abstraction removes details to help us focus our attention. For
instance, a car present a set of simple controls as an interface to its
drivers. As long as we know how to use the interface we can drive the
car without knowing how it operates under the hood. The internal working
of a car is unnecessary detail to drivers unless they are race car
drivers who need this type knowledges to drive the car most efficiently.
This interface hasn\'t change much since when the first car was made.
Abstraction also generalizes concepts by extracting common features from
specific examples. This car interface extracts common car features (what
a driver needs to know to drive a car) from all kinds of cars. Once you
learn how to drive one car you have learned how to drive all cars of the
same type. This is a powerful idea. Such abstraction also gives car
makers the freedom to change the internal design of a car without
affecting the users.

### Abstraction in Computing

We have learned that algorithm design is an integral part of problem
solving using computing and a first step in programming. The hardest
part isn\'t programming/coding but the keeping track of details in large
programs. There are two primary ways to keep our programs \"small\":
chunking and layering, which are two metaphors for abstraction. Chunking
breaks down (decompose) functionality into smaller units and let units
interact with each other through a well-defined interface. For instance,
in Snap! you can implement an algorithm as a block, which then can be
used anywhere in your script as long as you can call the block with a
proper sequence of parameters according to the interface. Layering
separates the functional units (blocks) into layers to separate concerns
and simply interactions patterns to make the complexity more manageable.
The following figure illustrates the idea of layering.

!By organizing functional units (blocks) into layers we can simply the
interactions and allow concurrent development of the
layers. into layers we can simply the interactions and allow concurrent development of the layers.")

In the figure each layer relies on the layer below it to function and
provides services to the layer above it. For example, a unit in layer
one is only allowed to call units in layer 2 below it. All interactions
are limited to pairs of layers that are next to each other in a stack of
layers. We could replace a layer completely with a new implementation
without affecting the rest of the stack, which achieves modularity. On
the contrary if any arbitrary interaction is allowed we may end up with
tightly coupled system as shown in the following figure.

!Without any restriction any unit (block) can call any other
unit. can call any other unit.")

### Abstraction Examples

Using abstraction to achieve simplification and generalization is truly
a powerful organizing idea. Recall the thought experiment in chapter
one, in which we built a machine that can potentially solve groups of
equations. The machine was built through abstraction - we construct
larger building blocks using smaller (more elementary) ones and treat
each block as a unit ignoring the internal details.

The Snap! environment allows us to construct programs in a similar
fashion. When a block is defined it becomes a new building block (user
defined). The block can be arbitrarily complex, but to use it we only
need to know the interface - its name and its list of parameters. The
unnecessary details are hidden from the users greatly simplifying the
thinking involved in programming.

Lets look at a concrete example. In order to make a sprite to draw
different equal-lateral shapes we can create a block for drawing each
type of shapes such as triangles, squares, pentagons, and etc. A block
in Snap! is an abstraction that hides details and represents a certain
behavior/intention. The following block draws a square with a specific
size.

!This Snap! block draws a square with a specific
size.
To draw a triangle we can use a similar logic structure.

!This Snap! block draws an equal-lateral
triangle.

The draw triangle block repeat three times to draw each side with a turn
of 120 degrees. Do you know why the sprite has to turn 120 degrees to
form a 60 angel between the two sides? I hope you have noticed that the
same logic structure can be used to create blocks for drawing any
equal-lateral shapes. Instead of creating a block for each shape we can
generalize the task into one block that can draw any shapes. This block
needs an additional piece of information - the number of sides.

!The Snap! block can draw any equal-lateral
shape.

With the number of sides we can determines the internal angle of the
shape, which is all we need to draw the shape. Please checkout this
resource
if you are not sure how to calculate the internal angle using the number
of sides. This block can serve as an abstraction of the tasks of drawing
equal-lateral shapes (polygons). You may have noticed the length of the
sides is hard-coded (typed in as a constant not a parameter). What if we
want to draw shapes of different size? We can further generalize the
function of the block by adding anther parameter and use it to control
the side length.

!This Snap! block draws any equal-lateral polygon of any
size.

Through this example, we have demonstrated defining blocks to abstract
away details of a task and generalizing a solution to solve more
problems.

### Recursion

Recursion is a pattern that
is self-similar - the whole consists of smaller parts that are
structurally similar to the whole. For example, a tree consist of
branches that look like smaller trees. Similarly, a directory tree of a
file system on a computer and an ancestry tree genealogy exhibit a
similar pattern. The following figure shows a recursive tree.

!Tree created using the Logo programming language and relying heavily
on
recursion.

Self-similarity allows us to define concepts that exhibit such a pattern
in a more concise and elegant way. A tree can be either a trunk with no
branches or a trunk with a number of branches, each of which is a tree.
This definition covers all possible tree structures. How would you
describe the following picture?

!`A visual form of recursion known as the Droste effect. The woman in this image holds an object that contains a smaller image of her holding an identical object, which in turn contains a smaller image of herself holding an identical object, and so forth.`

If you were to do it by delving into finer and finer details repeatedly
it never ends. Can you define the picture recursively? Another example
is the definition of the factorial function in math$$1!=1$$ and for all
$n>1 n! = n*(n-1)!$ This recursive definition not only defines factorial
but also describes a way to calculate factorial. For example, 5! can be
calculated from 4!, which is 4 times 3!, which is 3 time 2!, which is 2
times 1!, which is 1 by definition.

If the problem we are solving is self-similar - solving the whole
problem is a matter of solving the parts that are similar to the whole -
the solution we are defining for the whole can be used to solve the
subproblems(recursion). The beauty of a recursive solution is that the
definition of the problem is the solution as shown in the factorial
example. To design a recursive solution we practice wishful thinking -
as we describe the solution to whole problem we wish it is fully defined
and can be used to solve the smaller subproblems. In computer
programming, this is called recursive thinking/programming. To program
recursively, we describe the solution to a problem as a
function/procedure/block, in which we break the bigger problems into
smaller ones and use the function we defining to solve the smaller
problems. If the problem is finite, eventually the smaller problems are
so simple that they are directly solvable. In such cases the recursion
stops. We call those cases base cases. By the time our program reaches
all base cases, we would have solved the whole problem because all the
subproblems are solved including the problem we start with. In any
recursive function, two essential parts must exist: base cases and
recursive cases. The recursive cases part breaks a bigger problem into
smaller ones. The base cases stops the recursion by solving the directly
solvable problems. If both parts exists and are structured properly, the
algorithm (function) can solve problems of any size by asking clones of
itself to solve partial problems. Recursive problem solving is a
powerful idea because it simples our thinking: if you can define a
problem recursively you can solve it recursively. Recursive solutions
are more elegant and easy to verify, but it only lends itself to
problems that can be defined recursively.

Before we study some concrete examples we will introduce the concept of
function, which makes recursive solutions more manageable. A block in
Snap! is considered a function (similar to a math function) if it has
the following properties:

-   a function takes an arbitrary number of inputs (zero or more)
-   a function always returns/reports exactly one result value
-   for the same input a function always reports the same result value
-   the execution of a function has no side effects to the environment

With such restrictions functions in Snap! are blocks that performance a
task in isolation (in its own world) and hand off the result to be
further processed. According to the definition what blocks in the
following list are functions?

!Some blocks in this list are
functions.

### Recursion Examples

Consider the binary search algorithm:

`Find an item (target) in a sorted list`\
` if the list is empty, the target cannot be found`\
` consider the item in the middle, if it matches the target you are done `\
` otherwise, decide which half to search next`

To search one half of the ordered list is just a smaller version/part of
the overall problem so we can apply the same algorithm wishing the
algorithm is fully defined. This process goes on till a list is empty or
the search target is found.

Clearly the base cases are

-   if the list is empty, then the target can be found
-   if target in the middle of a list, then the target is found

The recursive cases are (assume the list is sorted in ascending order)

-   if the item in the middle of a list is smaller than the search
    target, continue searching in the second half of the list
-   otherwise, continue searching in the first half of the list