Planting Undetectable Backdoors in Machine Learning Models - Page 1
Forum section: electronics / general

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 28, 2023, 09:12:14 pm
content: Nice!https://arxiv.org/abs/2204.06974

author: TomKatt
author_rank: Frequent Contributor
timestamp: March 01, 2023, 12:51:40 pm
content: These AI type applications have become so complex, with datasets so large, that human minds can no longer understand their operation. And if you don't understand how something works, how would you ever really be able to identify errors or malicious activity?Quote from: Big Thinkhttps://bigthink.com/the-future/black-box-ai/

author: tszaboo
author_rank: Super Contributor
timestamp: March 01, 2023, 12:59:36 pm
content: So what is exactly the danger with this?As I understand this it would work the following: Having two models, one doing the expected operations, one the backdoor. So for example if you pass an image of a "blue stopsign next to a road" to an image classifier, object detector, it will tell you it's a cat. How can they exploit this?

author: tom66
author_rank: Super Contributor
timestamp: March 01, 2023, 03:01:03 pm
content: Quote from: TomKatt on March 01, 2023, 12:51:40 pmThis is the same issue with autonomous vehicles. The prediction by some was that we could build a neural network that takes an image in, runs it through the network, and generates steering, accelerator, brake, etc instructions. Unfortunately, that's effectively impossible to prove safe. The reality is that every successful autonomous vehicle out there uses the NN to process images, and solve the path and free space problem, but ultimately that data is processed by an ordinary algorithm running on an ordinary CPU, with that algorithm having been created by engineers sitting at their desks.

author: AndyC_772
author_rank: Super Contributor
timestamp: March 01, 2023, 03:18:24 pm
content: The example they give is about a hypothetical NN used to assess creditworthiness and approve or deny bank loans.The 'clean' NN would make a reasonable attempt to decide whether or not someone should be offered credit, based on the information available to the bank at the time.The 'backdoored' NN would, however, be made to unconditionally approve a loan, if some very particular, carefully crafted conditions were met. For example, a loan of $10,000 might be rejected, but a loan to the same individual of precisely $10,004.26 would be approved.Crucially, the agency that performed the training would be able to know what criteria would result in these 'false' approvals, but it would be mathematically very difficult to detect that they existed without that inside knowledge.

author: daqq
author_rank: Super Contributor
timestamp: March 01, 2023, 10:20:13 pm
content: Quote from: tszaboo on March 01, 2023, 12:59:36 pmOf the top of my head:You use a neural network to recognize bank notes and add a back door that if there's, say, a proper constellation of smudges, it'll recognize it as a bigger value?You train a model for a military device that's supposed to detect humans (for whatever reasons), but add a back door that if they are wearing a uniform with a particular design then it won't recognize them as humans?Then there's corporate sabotage - add some very obscure condition during which, say, a car won't recognize people as people? Good luck getting rid off the Blue Top Hat Killer Car nickname.