OpenLLM-France/wikimedia huggingface dataset:
# Data Coding Theory/Shannon capacity
## Channels

A channel is a communications medium, through which data can flow through. In a wireless network, the channel is the open space between the sender and the receiver through with the electromagnetic waves travel. In a wired network, the channel is the wire through with the electrical signals flow.

## Signal to Noise Ratio

Data flowing through a channel is subjected to outside noise interference. In a wireless system, this interference can be a competing transmittor, or even natural phenomena such as lightning or solar radiation. These same kinds of interference can affect wired transmissions as well.
The ratio, in dB (decibels) of the transmitted information power to the noise power in the received signal is known as the signal to noise ratio. The signal to noise ratio, abbreviated "SNR", is an important metric to learn when studying a transmission network, because it affects how much data can be sent and at what speed. If we have a signal with energy E, and a noise density of N, we calculate the SNR as:
    $SNR=10\log _{10}\left({\frac {E}{N}}\right)$
Likewise, if we have a signal with an amplitude of A, and a noise with variance σ, we calculate the SNR as:
    $SNR=10\log _{10}\left({\frac {A^{2}}{\sigma ^{2}}}\right)$
The higher the SNR, the more faithful the received signal will be to the sent signal. The lower the SNR, the more likely the received signal is going to contain errors. We won't cover SNR in too much detail in this book. However, the SNR is a very important quantity in a communications system, and there are classes of codes that are designed to help reduce errors in a channel with a low SNR.

## Bandwidth

Bandwidth is a measurement of a communication's channel, and determines what frequency ranges are transmitted faithfully through the channel, and which ranges are not. We don't need to discuss bandwidth in any detail in this book, but suffice it to say that it is an important quality of a communications channel.

## Shannon's Channel Capacity

The Shannon Channel Capacity is a formula that relates the bandwidth of a channel, the channel's SNR, and the bit rate.
[Shannon's Channel Capacity]
    $C=W\log _{2}(1+SNR)$
The channel capacity is the bit rate at which data can be sent along a channel with a negligible error rate. Attempting to send bits faster than this rate will increase the error rate beyond a negligible value. This will have large negative effects on our communications system. Notice that for a fixed bandwidth, as SNR decreases, the bitrate must also decrease.

## Ramifications of Channel Capacity

Because of Shannon's Channel Capacity, we can see a few important facts. The first of which is, the more noise a channel has, the slower our bitrate needs to be. This means that if we want to send data at or above the channel capacity, we will need to have a robust error checking and correcting mechanism.
What we can also see is that for a fixed SNR, more bandwidth means a faster data rate.
In short, to send more information over a channel more quickly, we need to:
* Increase the SNR
* Increase the bandwidth
* Use better coding and error correcting.
# Data Compression

There are many algorithms—and even more implementations of those algorithms—that accept unencoded (plain) information and encode it to use fewer bits. Perhaps even more important is the paired algorithm that accepts the encoded bits and extracts the information.
Each pair of algorithms—one that creates the encoded form, and the other that accepts the encoded form and extracts the information—is called a data compression algorithm. These type of algorithms are increasing abundant, as are their variations, most utilize dictionary based schemes and statistical methods. They are also becoming increasingly specialized in compressing specific kinds of data—text, speech, music, photos, or video...
Data compression is useful in some situations because "compressed data" will save time (in reading and on transmission) and space if compared to the unencoded information it represent. Many programmers attempt to develop new algorithms to tightly compress the data into as few bits as possible (while still being able to recover the relevant information). However, other programmers—after testing several algorithms—deliberately pick some algorithm *other* than the one that would give them the fewest compressed bits. They deliberately sacrifice a few bits in order to improve latency, reduce compression time, or reduce decompression time.
Some communication systems carefully and deliberately add small amounts of "redundancy"—the Data Coding Theory Wikibook has more details.
There are also several data compression benchmarks available for comparing data compression algorithms—even one 50,000 euro cash prize for compressing one particular benchmark file as small as possible (and, of course, uncompressing it afterwards).
In this book, we describe the decompressor first, for several reasons:
* Decompression routines are usually much easier to understand than the paired compression routine.
* Too many people write the compressor first, then later discover that it doesn't save enough information to reconstruct the original file.
* Once we have a decompressor, we can often improve the compressor alone without changing the file format or the decompressor.
* Many compression systems only document and standardize the decompressor and the file format. (Probably because of the above reasons).

## Contents

* Coding
* Evaluating Compression Effectiveness
* Models
** Markov models
** The zero-frequency problem
** data differencing
* Dictionary compression
* grammar-based compression
* Order/Entropy
* Lossy vs. Non-Lossy Compression
* Inference Compression
* Streaming Compression
* compressed file systems
* Asymmetric Compression
* Multiple transformations
* Executable Compression
* References

## References

* ↑ "Compression Basics" by Pasi 'Albert' Ojala 1997
# Data Coding Theory/Transmission Codes
## Basic Coding

It turns out that we can actually improve the performance of our system simply by changing the bits in our system. We can change the order of bits, or we can change the number of bits that we send, to help decrease error rates. This chapter will talk about transmission codes that can help increase system performance.

## Gray Codes

Let us look at a basic QASK system:
* "00" = +5V
* "01" = +1.66V
* "10" = -1.66V
* "11" = -5V
We will call these "symbol 1" (for 00), "symbol 2" (for 01), "symbol 3" (for 10) and "symbol 4" (for 11).
Now, let us make a basic assumption that the only errors we get will be errors between adjacent symbols. For instance, Symbol 1 can be mistaken for symbol 2, but symbol 1 can never be mistaken for symbol 3. In reality, there is always a very small chance that this could happen, but we will simply ignore it for our purposes.
Now, we have 6 error conditions:
* Symbol 1 looks like Symbol 2
* Symbol 2 looks like Symbol 1
* Symbol 2 looks like Symbol 3
* Symbol 3 looks like Symbol 2
* Symbol 3 looks like Symbol 4
* Symbol 4 looks like Symbol 3
We can also see that in error conditions 1, 2, 5, and 6, a symbol error only produces 1 bit error. The other bit is actually not transmitted in error. The only time where we see 2 bits transmitted in error is when there is a mixup between symbols 2 and 3. We can calculate our probability of bit error as such:
$P_{\mbox{bit error}}={\frac {4}{6}}(1)P_{\mbox{one bit error}}+{\frac {2}{6}}(2)P_{\mbox{two bits error}}$
P$_{onebiterror}$ is the probability that 1 bit from the symbol will be in error. P$_{twobitserror}$ is the probability that both symbols will be received in error.
To alleviate this problem, we can reorder our symbols, and map them the following way:
* "00" = +5V
* "01" = +1.66V
* "11" = -1.66V
* "10" = -5V
Now, when any symbol is received in error, there is only one bit error. This small trick is called "Gray Coding", and is a useful (but simple) method to improve system performance in terms of net bit errors.

### Using Gray Codes

Gray codes are a simple example of how using an encoding/decoding mechanism in a communications system can have an effect on the error rate of the system. In future chapters we will discuss error correction codes (ECC) that can be used to find and fix single-bit errors in a signal. If we use Gray codes to ensure that the majority of errors are single-bit errors, we can then use more advanced ECC codes to resolve these errors.
Gray codes are typically not used on their own, but are combined with more robust coding methods, like described above.

## For further reading

* Wikipedia:Manchester codes
* Wikipedia:MLT-3 Encoding
# Economic Sophisms/182

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"government had these roads accurately measured; and then it announced to the agriculturist, 'All that you can steal from travellers between these two points is yours; let that serve as a premium for your protection and encouragement.' Afterwards it assigned to each manufacturer, to each shipowner, a certain portion of road, to be made available for their profit, according to this formula:—
Template:Fs90/s
* Dono tibi et concedo
* Virtutem et puissantiam
* Template:GapVolandi,
* Template:GapPillandi,
* Template:GapDerobandi,
* Template:GapFiloutandi,
* Template:GapEt escroquandi,
* Impune per totam istam
* Viam."
Template:Fs90/e
Now it has come to pass tliat the natives of the kingdom of A. have become so habituated to this system, that they take into account only what they are enabled to steal, not what is stolen from them, being so determined to regard pillage only from the standpoint of the thief, that they look upon the sum total of individual thefts as a national gain, and refuse to abandon a system of protection, without which they say no branch of industry could support itself.
You demur to this. It is not possible, you exclaim, that a whole people should be led to ascribe a redundancy of wealth to mutual robbery.
And why not? We see that this conviction pervades France, and that we are constantly organizing and improving the system of reciprocal robbery under the respectable names of premiums and protective tarifis.
We must not, however, be guilty of exaggeration. As regards the mode of levying, and other collateral circumstances, the system adopted in the kingdom of A. may be worse than ours; but we must at the same time admit that, as regards the principle and its necessary consequences, there is not an atom of difference between these two species of theft; which are both organized by law for the purpose of supplementing the profits of particular branches of industry.
# Economic Sophisms/184

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"Template:Hwe that all which an individual steals from the musses is a general gain. Perpetual motion, the philosopher's stone, the quadrature of the circle, are antiquated problems; but the theory of progress by plunder is still held in honour. A priori, we should have thought that, of all imaginable puerilities, it was the least likely to survive.
Some people will say. You are partisans, then, of the laissez passer?— economists of the school of Smith and Say? You do not desire the organization of labour. Yes, gentlemen, organize labour as much as you choose, but have the goodness not to organize theft.
Another, and a more numerous, set keep repeating, premiums, tariffs, all that has been exaggerated. We should use them without abusing them. A judicious liberty, combined with a moderate protection, that is what discreet and practical men desire. Let us steer clear of fixed principles and inflexible rules.
This is precisely what the traveller tells us takes place in the kingdom of A. "Highway robbery," say the sages, "is neither good nor bad in itself; that depends upon circumstances. All we are concerned with is to weigh things, and see our functionaries well paid for the work of weighing. It may be that we have given too great latitude to pillage; perhaps we have not given enough. Let us examine and balance the accounts of each man employed in the work of pillage. To those who do not earn enough, let us assign a larger portion of the road. To those who gain too much, we must limit the days or months of pillage."
Those who talk in this way gain a great reputation for moderation, prudence, and good sense. They never aspire to the highest offices in the state.
Those who say, Repress all injustice, whether on a greater or a smaller scale, suffer no dishonesty, to however small an extent, are marked down for ideologues, idle dreamers, who keep repeating over and over again the same thing. The people, moreover, find their arguments too clear, and why should they be expected to believe what is so easily understood?
# Economic Sophisms/18

<pagequality level="4" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"department would supply the whole consumption of France?" Then, in his eyes, good lies in sterility, in dearth, and evil in fertility and abundance.
The Presse, the Commerce, and the greater part of the daily papers, have one or more articles every morning to demonstrate to the Chambers and the Government, that it is sound policy to raise legislatively the price of all things by means of tariffs. And do the Chambers and the Government not obey the injunction? Now tariffs can raise prices only by diminishing the supply of commodities in the market! Then the journals, the Chambers, and the Minister, put in practice the theory of scarcity, and I am justified in saying that this theory is by far the most popular.
How does it happen that in the eyes of workmen, of publicists, and statesmen, abundance should appear a thing to be dreaded, and scarcity advantageous? I propose to trace this illusion to its source.
We remark that a man grows richer in proportion to the return yielded by his exertions, that is to say, in proportion as he sells his commodity at a higher price. He sells at a higher price in proportion to the rarity, to the scarcity, of the article he produces. We conclude from this, that, as far as he is concerned at least, scarcity enriches him. Applying successively the same reasoning to all other producers, we construct the theory of scarcity. We next proceed to apply this theory, and, in order to favour producers generally, we raise prices artificially, and cause a scarcity of all commodities, by prohibition, by restriction, by the suppression of machinery, and other analogous means.
The same thing holds of abundance. We observe that when a product is plentiful, it sells at a lower price, and the producer gains less. If all producers are in the same situation, they are all poor. Therefore it is abundance that ruins society. And as theories are soon reduced to practice, we see the law struggling against the abundance of commodities.
This sophism in its more general form may make little impression, but applied to a particular order of facts, to a certain branch of industry, to a given class of producers, it is extremely specious; and this is easily explained. It forms a
# Economic Sophisms/186

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"Template:Hwe who educates them, and, lastly, your humble servant, who cannot be expected to work exactly for nothing.
J.: All right; service for service is quite fair, and I have nothing to say against it. I should like quite as well, no doubt, to deal directly with the rector and the schoolmaster on my own account; but I don't stand upon that. This accounts for the second tun—but we have still other four to account for.
L.: Would you consider two tuns as more than your fair contribution to the expense of the army and navy?
J.: Alas! that is a small affair, compared with what the two services have cost me already, for they have deprived me of two sons whom I dearly loved.
L.: It is necessary to maintain the balance of power.
J.: And would that balance not be quite as well maintained if the European powers were to reduce their forces by one-half or three-fourths? We should preserve our children and our money. All that is requisite is to come to a common understanding.
L.: Yes; but they don't understand one another.
J.: It is that which fills me with astonishment, for they suffer from it in common.
L. : It is partly your own doing, Jacques Bonhomme.
J.: You are joking, Mr Taxgatherer. Have I any voice in the matter?
L.: Whom did you vote for as deputy?
J.: A brave general officer, who will soon be a marshal, if God spares him.
L.: And upon what does the gallant general live?
J.: Upon my six tuns, I should think.
L.: What would happen to him if he voted a reduction of the army, and of your contingent?
J.: Instead of being made a marshal, he would be forced to retire.
L.: Do you understand now that you have yourself …
J.: Let us pass on to the fifth tun, if you please.
L.: That goes to Algeria.
J.: To Algeria! And yet they tell us that all the Mussulmans are wine-haters, barbarians as they are! I have often inquired whether it is their ignorance of claret which has made them infidels, or their infidelity which has made them ignorant
# Economic Sophisms/180

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"Template:Hwe. Let us cultivate union as regards that which constitutes our strength. If we give up a single point of the theory of protection, the whole theory falls to the ground.
"SHIPOWNER: For us shipowners protection has been a failure. I repeat that the merchant marine is at its lowest ebb.
"SHIPMASTER: Well, let us raise the surtaxe, and let the shipowner who now exacts 30 francs from the public for his freight, charge 40.
"A MINISTER; The government will make all the use they can of the beautiful mechanism of the surtaxe; but I fear that will not be sufficient.
"A GOVERNMENT FUNCTIONARY: You are all very easily frightened. Does the tariff alone protect you? and do you lay taxation out of account? If the consumer is kind and benevolent, the taxpayer is not less so. Let us heap taxes upon him, and the shipowner will be satisfied. I propose a premium of five francs to be levied from the public taxpayers, to be handed over to the shipbuilder for each ton of iron he shall employ.
"Confused voices: Agreed! agreed! An agriculturist: Three francs premium upon the hectolitre of corn for me! A manufacturer: Two francs premium on the yard of cloth for me! etc., etc.
"THE PRESIDENT: This then is what we have agreed upon. Our session has instituted a system of premiums, and it will be to our eternal honour. What branch of industry can possibly henceforth be a loser, since we have two means, and both so very simple, of converting our losses into gains—the tariff and the premium? The sitting is adjourned."
I really think some supernatural vision must have foreshadowed to me in a dream the near approach of the premium (who knows but I may have first suggested the idea to M. Dupin?) when six months ago I wrote these words:—
"It appears evident to me that protection, without changing its nature or the effects which it produces, might take the form of a direct tax, levied by the state, and distributed in premiums of indemnification among privileged branches of industry."
And after comparing a protective duty to a premium, I added,
"I confess candidly my preference for the last system. It
seems to me juster, more economical, and more fair. Juster,
# Economic Sophisms/183

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"Remark also, that if highway robbery presents some inconveniences in its actual perpetration, it has likewise some advantages which we do not find in robbery by tariff.
For example, it is possible to make an equitable division among all the producers. It is not so in the case of customs duties. The latter are incapable of protecting certain classes of society, such as artisans, shopkeepers, men of letters, lawyers, soldiers, labourers, etc.
It is true that the robbery by premium assumes an infinite number of shapes, and in this respect is not inferior to highway robbery; but, on the other hand, it leads frequently to results so whimsical and awkward that the natives of the kingdom of A. may well laugh at us.
What the victim of a highway robbery loses, the thief gains, and the articles stolen remain in the country. But under the system of robbery by premium, what the tax exacts from the Frenchman is conferred frequently on the Chinese, on the Hottentots, on the Caffres, etc., and here is the way in which this takes place:
A piece of cloth, we shall suppose, is worth 100 francs at Bordeaux. It cannot be sold below that price without a loss. It is impossible to sell it above that price because the competition of merchants prevents the price rising. In these circumstances, if a Frenchman desires to have the cloth, he must pay 100 francs, or want it. But if it is an Englishman who wants the cloth, the government steps in, and says to the merchant, "Sell your cloth, and we will get you 20 francs from the tax-payers." The merchant who could not get more than 100 francs for his cloth, sells it to the Englishman for 80. This sum, added to the 20 francs produced by the premium theft, makes all square. This is exactly the same case as if the taxpayers had given 20 francs to the Englishmen, upon condition of his buying French cloth at 20 francs discount, at 20 francs below the cost of production, at 20 francs below what it has cost ourselves. The robbery by premium, then, has this peculiarity, that the people robbed are resident in the country which tolerates it, while the people who profit by the robbery are scattered over the world.
Verily, it is marvellous that people should persist in Template:Hws
# Economic Sophisms/188

<pagequality level="3" user="Zoeannl" />style="background: #ececec; text-align: left; padding-left: 0.5em; font-weight: bold;" class="table-rh"making him my proxy, I was guilty of a piece of folly; for what is there in common between a general officer and a poor vinedresser?
L.: Oh, yes; there is something in common, namely, the wine, which he has voted to himself in your name.
J.: You may well laugh at me, Mr Taxgatherer, for I richly deserve it. But be reasonable. Leave me at least the sixth
tun. You have already secured payment of the interest of the debt, and provided for the civil list and the public service, besides perpetuating the war in Africa. What more would you have?
L.: It is needless to higgle with me. Communicate your views to Monsieur le General, your representative. For the present, he has voted away your vintage.
J.: Confound the fellow! But tell me what you intend to make of this last cask, the best of my whole stock? Stay, taste this wine. How ripe, mellow, and full-bodied it is!
L.: Excellent! delicious! It will suit Mons. D., the cloth-manufacturer, admirably.
J.: Mons. D., the cloth-manufacturer? What do you mean?
L.: That he will reap the benefit.
J.: How? What? I'll be hanged if I understand you!
L.: Don't you know that Mons. D. has set on foot a grand undertaking, which will prove most useful to the country, but which, when everything is taken into account, causes each year a considerable pecuniary loss?
J.: I am sorry to hear it, but what can I do?
L.: The Chamber has come to the conclusion that, if this state of things continues, Mons. D. will be under the necessity of either working more profitably, or of shutting up his manufacturing establishment altogether.
J.: But what have these losing speculations of Mons. D. to do with my wine?
L.: The Chamber has found out that, by making over to Mons. D. some wine taken from your cellar, some corn taken from your neighbour's granaries, some money kept off the workmen's wages, the losses of that enterprising patriot may be converted into profits.
J.: The recipe is as infallible as it is ingenious. But, zounds!