Q:

Why did Aegon Targaryen give up attempting to conquer Dorne? 

The Targaryen army withdrew from Dorne, leaving the country untaken.  A thousand pinpricks did what a single pitched battle could not, and Aegon chose to withdraw and leave Dorne an independent nation. 
A Wiki of Ice and Fire, War of
  Conquest

Why did Aegon give up?  What kind of tactics were used by House Martell to make this happen? 

A:

It wasn't that Aegon didn't want to capture it.  He couldn't, because the defendants fought in a way Aegon could not use his dragons.  They knew that they didn't stand a chance in open battle.  They were using ambush techniques and were not hiding in castles like other kings did. 
Whereas other kings and lords had taken to the field against Aegon, or clustered in castles, the Dornish refused to give open battle and allow Aegon to deploy his dragons.  Instead, they turned to ambush and raids, striking quickly and then slipping back into the desert or through the mountain passes, where even the dragons could not find them.  In time, Aegon pulled away from Dorne.  
(A Dance with Dragons, Chapter 8, Tyrion & A Feast for Crows, Chapter 28, Cersei)

A:

Accepted answer is correct but lacking in details.  
This has been discussed in detail in the new book The World of Ice and Fire.  Since when previous answer was given, this information had not been revealed so I want to add it here.  
I will split the answer in different parts as per important events in the failed invasion of Dorne.  
Rhaenys' Mission
Quasi-Invasion of Dorne and Vale started at proximately the same time.  Visenya rode Vhagar to Vale and Rhaenys rode Meraxes to Dorne to negotiate their surrender and give them a last chance of submitting before unleashing the dragons on them.  It was Quasi-invasion because the expected outcome was submission of the said Kingdoms after show-of-power in form of Dragons flying over their heads but at the same time, these missions were unaccompanied by traditional invading forces.  
Visenya won a bloodless victory when she took King of the Vale to fly with her on her dragon. 
Rhaenys did not have such an easy time however.  From Land route, there are two ways of entering Dorne.  

The Prince's pass in Red Mountains connecting Dorne to the Reach
The Boneway pass in Red Mountains, connecting Dorne to Stormlands.  

Rhaenys took the Prince's pass route.  She saw a Dornish Army guarding the pass but she did not engage them and instead decided to fly directly to castles of the Lords and force their submission without shedding blood.  
Rhaenys first flew to Vaith but found the Castle empty.  She then moved to Godsgrace but that castle was abandoned as well.  Upon inquiring, Peasants only told her that Lords were gone to an undisclosed location.  Rhaenys then moved East to Planky Town, chief port of Dorne.  She found only women and children there and all the men were gone.  
At that Rhaenys decided to visit Sunspear directly.  When she reached there, Sunspear was empty as well but the Princess of Dorne, Meria Martell was present there.  Following conversation took place between the Princess and the Queen:

“Meria: I will not fight you, nor will I kneel to you.  Dorne has
  no king.  Tell your brother that. 
Rhaenys: I shall, but we will come again, Princess, and next time we shall come with fire and blood. 
Meria: Your words, Ours are Unbowed, Unbent, Unbroken.  You may burn us, my lady, but you will not bend us, break us, or make us bow. 
  This is Dorne.  You are not wanted here.  Return at your peril." 

Meria made it clear for Rhaenys that they had no intention to fight the Targaryens but they will never submit either.  At that Rhaenys went back to her brother who had just taken Old Town and was crowned there as "King of the Andals and The Rhoynar and the First-men, Lord of the Seven Kingdoms".  So as evident by the titles, Aegon certainly wanted to conquer Dorne as he proclaimed himself King of the Rhoynar and Lord of the Seven Kingdoms.  
Pictured, Queen Rhaenys meeting Princess Meria Martell at Sunspear. 

First Dornish War
Aegon left Dorne alone for sometime while he established his rule in King's Landing.  
He was done with those administrative tasks and minor wars in four years so in 4 A.C. he began full invasion of Dorne to finish the conquest.  
Dornishmen resorted to fighting a guerrilla war instead of facing the Targaryens in a pitched battle.  They abandoned their castles since they knew what happened to King Harren the Black in burning of Harrenhal and fled into Dornish wilderness which was composed mountainous and desert terrains, perfect for this sort of fighting.  Not to mention, Dornishmen were fighting on home ground while invaders were fighting in hostile terrain with terrible weather for those unaccustomed to heat.  
Rhaenys' assault on Eastern coastline of Dorne
Rhaenys struck the first blow.  She flew along the coastline and struck at Planky Town.  She burnt the whole city and lead her men Northwards to Sunspear.  I am not sure about the type of force she commanded but it was presumably an amphibious force because her mission was to attack Dorne from sea and strike towards Sunspear. 
Aegon's attack from Prince's Pass
In the Meanwhile, Aegon lead an army from Prince's Pass into Dorne.  Lords of Red Mountain did not offer regular resistance to Aegon.  As Aegon was making his way to Hellholt, Lords of Red Mountain performed hit and run attacks on Invaders, causing huge damages.  Whenever Aegon or Visenya mounted their dragons, the Dornish Fighters disappeared in the terrain.  When finally Aegon reached Hellholt, he found that House Uller who ruled the land were all gone before he could capture the castle. 
Annoyed, Aegon moved North-West to Yronwood.  He took the castle but was further annoyed when he found that Castle was defended by old men and children while fighting men were all gone.  When Aegon reach Skyreach, he found that the castle of house Fowler was abandoned.  So Aegon considered his quest of conquering Red Mountains as completed and move Eastwards along the coast to take Ghost hill, Stronghold of House Toland.  
As soon as Aegon besieged Ghost Hill, a challenger in livery of House Toland challenged Aegon to single combat.  Aegon made short-work of the challenger, expecting him to be Lord Toland.  However, when the corpse was identified, It was revealed that the Challenger was actually fool of Lord Toland while the Toland Family had already fled.  
Orys Baratheon's advance from Boneway
Aegon had planned to invade Dorne from all sides.  So command to attack from Boneway was Orys Baratheon's.  Orys suffered a disastrous defeat when Dornish men attacked the Targaryen Army in a surprise Night attack.  Targaryens were trapped.  Many notable Lords were taken POWs by the Dornish which included Hand of The King, Orys Baratheon, The man rumoured to be bastard brother of Aegon, Rhaenys and Visenya (Confirmed in Fire and Blood).  
Dragon banner flies atop Sunspear
Still, save the assault from Boneway, the other two Targaryen invading forces were advancing successfully.  There was little to no resistance as castles were abandoned and soldiers were vanished.  Soon, Rhaenys and Aegon joined each other at Sunspear where they found the seat of House Martell empty.  Aegon declared himself as victor of the Dornish war and named Lord Rosby as Royal Viceroy of Sunspear.  He also stationed an Army in Dorne under command of Lord Tyrell to put down any revolts as Aegon knew that power of House Martell was still intact as they had not fought in any major confrontation.  
Content with the outcome and confident about Lord Tyrell's capabilities, Aegon and his sisters left Dorne for King's Landing.  
Unbowed, Unbent, Unbroken - Dornish Revolt
As soon as Dragons left, Rebellion spread like wildfire as that was the Dornish plan all along.  Martell Forces rallied and the war finally began in earnest.  
Lord Tyrell and his entire army disappeared somewhere between Hellholt and Vaith, presumably destroyed by Martells.  
Victorious Martells then moved to liberate Sunspear.  The Targaryen Viceroy (Officially Warden of the Sands), Lord Rosby was arrested.  Dornishmen took him to the roof of Spear Tower and blind eighty years old Meria Martell personally threw him down to his death.  
Treachery of the Dornishmen
Martells offered to free the POWs for ransom.  Aegon consented as his Hand, Best Friend and bastard half-brother Orys Baratheon was among the captives.  Dornishmen demanded weight of each man in gold as ransom.  Aegon agreed and ransom was paid.  However, Dornishman Lord Wyl cut off sword hands of the captives before freeing them (Orys eventually avenged himself on Lord Wyl's son by taking his sword hand as payment and the other hand and both feet as his usury).  
Aegon's Revenge
Aegon was enraged.  Dornishmen had broken the agreement of release of Prisoners by mutilating them.  Aegon and his sisters burnt many Dornish cities in revenge.  In retaliation for Air-attacks, Dornishmen launched raids in Reach and Stormlands.  Aegon burnt Starfall, seat of House Dayne.  Rhaenys went to burn Hellholt, seat of House Uller.  Hellholt was burnt but an ironbolt from a scorpion pierced eye of Queen Rhaenys' dragon Meraxes and she fell.  She was presumed dead as she took quiet a tumble there.  
Fire and Blood - Dragon's Wroth
Aegon, Orys and Visenya were devastated by death of Rhaenys.  The next two years were called Dragon's wroth due to wanton brutality and destruction showed by Targaryens to avenge Rhaenys.  
Aegon and Visenya burnt every Dornish castle except Sunspear.  Some people say Aegon and Visenya had come to fear that Princess Meria had somehow figured out how to bring down Dragons therefore they avoided attacking Sunspear.  An alternate reason is that Aegon had hoped if Dorne was to burn and Sunspear was to remain intact, it would turn Dornishmen against Martells.  Aegon launched a campaign of Propaganda that Martells had actually paid the Targaryens to spare their castle while decimating rest of Dorne.  Dornishmen however rejected this propaganda and remained loyal to Martells.  
Murder attempts and bounties
Still unsatisfied, Aegon and Visenya declared the Dornish lords to be rebels and outlaws and placed bounties on their heads.  Dornish responded by placing bounties on heads of the Targaryen siblings.  
Aegon and Visenya came very close to be assasinated but Visenya's quick thinking and sword skills saved them.  Then Visenya founded the order of King's Guard to protect the King from Dornish assassins.  
In the meanwhile, Martells committed atrocities in Stormlands.  
Peace and Status Quo Ante Bellum
After 9 years of warfare, Princess Meria Martell died.  A Man of peace, Her Son, Prince Nymor Martell sent a peace delegate under his daughter the Princess Deria.  He also sent skull of Meraxes though it is unclear whether it was sent as a warning or as a gesture of goodwill.  
Nevertheless, The Skull was seen as an open threat in King's Landing and response was outrage specially by powerful leaders such as Orys Baratheon and Visenya Targaryen.  Aegon was about to refuse Deria's peace terms but then the Princess produced a letter from her father which was for the King's Eyes only.  
No one knows the contents of the letter but Aegon's color fled when he read it and he cut himself on the throne.  Some people suggest that the letter said Rhaenys was alive but less than dead.  Martells had tortured Rhaenys into insanity and offered to end her misery and return her corpse if Aegon was to give them peace.  
Aegon left King's Landing for sometime, presumably to receive corpse of Rhaenys.  When he returned, he signed the peace agreement with the Dornish.  

Reasons of Failure and Abandoning of the invasion
Reasons of failure should be evident from the complete account of the invasion.  In any case, the reasons of Aegon's failure and lack of further attempts appear to be:

Unity of Dornishmen against Dragon invasion: Rhoynar resented Dragons and Dragonlords since it was Valyrian dragonlords who forced them to flee from Rhoyne and take refuge in Dorne.  They were united under leadership of their Princess Meria.  
Avoiding Major confrontations: Dornishmen learnt lessons from campaigns of other Kings of Westeros and wisely concluded that open confrontation would never play to their benefit.  So they conserved their forces instead of fighting a pitched conclusive battle against Targaryens.  
Guerrilla attacks:  Dornishmen employed Guerrilla tactics against the invaders, attacking hard and then running fast before a coordinated response could be mounted.  This affected morale of Targaryen troops and inflicted severe damages.  Targaryens were unready to fight this kind of warfare as their entire experience had been based on pitched conclusive battles with close support from Air.  
Hostile terrain and climate: Dornish terrain was mostly desert and mountainous.  This terrain is ideal for guerrilla tactics and Dornish used it to the best possible extent.  Dornish weather was hot and sunny which is more lethal for foreigners than it was for natives.  
Abandonment of fixed defenses: Dornish relied on mobile defense rather than defending castles.  They abandoned castles as they knew castles were vulnerable from air.  
Capacity to mount retaliatory attacks: Dornishmen always exacted toll on Targaryen lands for every attack on Dorne.  They attacked neighboring regions of Stormlands and the Reach, making Targaryens mindful of the price they would have to pay for any attack.  Targaryens failed to crush the fighting power of Martells and thus were forced to suffer retaliation.  
Natural defenses: Dorne was guarded by natural defenses in form of the Red Mountains which guarded the entry points into Dorne.  Due to mountainous terrain, Martells successfully attacked the invaders and their supply lines from Mainland Westeros.  
Stubborn Resistance: Despite incurring very heavy losses, Dornishmen were ready to defend their land till the end.  They simply never gave up even after almost a decade of fighting.  
Targaryen inability to use Dragons: Targaryens could not use their dragons efficiently as the enemy always retreated on first sight of the Dragons.  Until then, Dragons had won the wars for them.  But in Dornish war, that was of no avail since they did not know where could they find concentration of enemy troops.  They did try to force Dorne into submission by following a scorched earth policy but that did not work out as they failed to destroy the war machine of Dorne.  
Peace Pact with Martells: After 9 years of fighting, Aegon signed a peace treaty with the Dornish for undisclosed reasons.  Aegon was an honorable man and kept the pact throughout the rest of his reign although skirmishes with Dornish continued in reigns of his Sons Aenys and Maegor.  Later one of his descendants Daeron I Targaryen successfully forced Martells to surrender.  Dorne however soon won back their freedom and killed Daeron I in a treacherous attack during negotiations, further fanning the flames of animosity.  Later King Aegon IV also tried to mount a failed invasion of Dorne.  The wars finally ended when Aegon IV's son Daeron II brought Dorne to Targaryen fold through marriage pacts and Aegon's conquest was finally completed, centuries after his death. 