You only have to pay Capital Gains Tax on your overall gains above your tax-free allowance (called the Annual Exempt Amount).
The Capital Gains tax-free allowance is:
£12,300
£6,150 for trusts
You can see tax-free allowances for previous years.
You may also be able to reduce your tax bill by deducting losses or claiming reliefs - this depends on the asset.