Our Idiot Brother

This article's plot summary may be too long or excessively detailed. Please help improve it by removing unnecessary details and making it more concise. (January 2013) (Learn how and when to remove this template message)
Ned Rochlin (Paul Rudd) is a biodynamic farmer living with his girlfriend, Janet (Kathryn Hahn). While selling produce at a local market, Ned sells marijuana to a uniformed police officer due to the officer's claim that he had a stressful week, which made Ned feel sympathetic after initially being skeptical. Ned is then arrested on a charge of selling drugs.
Ned has three sisters: Miranda (Elizabeth Banks) is a journalist for Vanity Fair trying to get her first major article published. Though she has trouble finding a man to keep her interest, she and a neighbor, Jeremy (Adam Scott) have hidden feelings for each other. Natalie (Zooey Deschanel), is an independent, bisexual hipster living with her girlfriend, Cindy (Rashida Jones), and five other roommates. Liz (Emily Mortimer), the oldest, is married to Dylan (Steve Coogan), a documentary filmmaker. Their marriage is failing as Dylan shows no sexual or emotional interest in Liz. They also have strict control over their son River (Matthew Mindler), which leaves him unhappy and unable to express himself.
When Ned is released from prison, he returns home to his girl and his dog ("Willie Nelson"). He finds that she is living with Billy (T. J. Miller), and no longer wishes to continue their relationship or allow him to work at the farm. Billy gives Ned a ride into town and tells him that if he can scrape together $500, Janet might let him stay in the goat barn for rent on the farm. Ned stays at his mother's house which he is less than enthusiastic about. A few days later, Ned shows up at Liz's house, asking if he can stay with her. He is put in River's room and told that he must help around the house and work with Dylan on his newest documentary about a Russian ballerina named Tatiana (Lydia Haug).
The next day, Miranda reluctantly asks Ned to chauffeur while she interviews an important client, Lady Arabella (Janet Montgomery). Miranda hopes to pry into her scandalous past, but is dismayed to learn of a legal agreement to only ask about charity work. Miranda tries to pretend Ned isn't there, but his friendliness charms Arabella, who takes a liking to him. That night, Ned goes with Natalie to a self-help meeting with Natalie's artist friend, Christian (Hugh Dancy). Christian is attracted to Natalie but is dissuaded by her lesbian relationship until Ned informs him that she is bisexual and likes guys too. Christian and Natalie take a cab home and end up having sex.
Ned goes back to working with Dylan, but is told to watch the car while Dylan conducts a private interview with Tatiana. Hours later, Ned is told by a police officer to move out of a tow zone. He runs upstairs to get the keys and discovers Dylan naked with Tatiana. Dylan later says the only reason he was naked was to make Tatiana feel more "comfortable" while being naked which Ned believes. The next night, Ned goes with Miranda to Arabella's benefit dinner, and stays behind to have a conversation with Arabella. She explains the gossipy part of her life (that Miranda had hoped to discover) to Ned, who listens with interest. Ned is later kicked out of Liz's house because, at an important school interview, River tells the interviewer that Ned just got out of jail for selling drugs to a policeman. Ned apologizes, stating he was only trying to help River be himself. Now staying with Miranda, Ned casually mentions Dylan's nude interview with Tatiana and Miranda concludes that Dylan is having an affair. Ned also mentions a personal detail about Arabella that makes Miranda realize that Ned has the info she needs. She forces Ned to tell her the details, which Ned is ashamed to do since he was trusted. Ned returns to the farm with money but is not allowed to rent the barn.
Ned asks Cindy, who is a lawyer, about ways to get custody of his dog but she suggests as an alternative that they work together to steal the dog from Janet. He later goes out to lunch with Jeremy and hints at Jeremy's attraction towards his sister. Jeremy explains that Miranda is too bossy for him, though admitting she is beautiful. Ned mentions the idea to Miranda, who is hurt to learn what Jeremy had to say and tells Ned her thoughts on Jeremy, leading to a fight between the two. Natalie later confides in Ned that she is pregnant. Miranda and Nat try to tell Liz about Dylan's affair but Liz becomes angry and the conversation turns into an argument between the three about their personal lives. Liz later confronts Dylan about the affair, and he blames her attempts at pleasing him.
Miranda takes Ned into work the next day to vouch for his statements. Ned reads the article and is shocked to see how personal Miranda made it and refuses to admit that it was all true in order to spare Arabella the embarrassment to the company's lawyer. However, this means Miranda's article will not get published and her editor is angry. She kicks Ned out and he stays with Nat, who lies about having told Cindy about the cheating. The next morning, Cindy and Ned drive to the farm to steal his dog back. They sneak into the house but Ned casually brings up Nat's infidelity and Cindy angrily calls Nat from inside the house, which alerts Janet to their presence. Janet refuses to give Ned his dog and Cindy drives away, leaving Ned behind.
Ned goes to his parole officer, Omar, and tells him that the stress of his life was getting to him and he smoked marijuana with a neighbor. Omar tells Ned now he must report him. Ned goes home to family dinner and is blamed by his sisters for all the trouble in their lives. Ned finally loses his temper and yells at them for their selfishness and for bringing everyone else down which seems to strike a chord with them. Omar arrives with an officer to take Ned into custody despite being upset at having to do so. Miranda posts Ned's bail, but he refuses to sign his release form, opting to stay in prison rather than be with the girls. They devise a plan to get Willie Nelson from Janet in order to get him to leave. Although Janet refuses, Billy comes out of the house carrying Willie Nelson, tired of Janet refusing to give Ned his dog. They take Willie Nelson to the prison where he is reunited with Ned, motivating him to leave prison.
A few weeks later, Ned is out to lunch with his sisters. Nat receives a call from Cindy, who wants to go with Nat to her gynecologist appointments for support. Miranda has patched things up and started a relationship with Jeremy. Liz has decided to start dating again and has started letting River be himself. Meanwhile, Ned and Billy have opened up a small homemade candle shop together. One day, Ned cannot find Willie Nelson and begins running through town looking for him. He finds Willie playing with another dog who has also run away from her owner. The owner, Amy, comes up and when Ned asks her dog's name, the owner replies "Dolly Parton" to which Ned says his dog is "Willie Nelson" and smiles.

Question: Who refuses to give Ned his dog back?
Answer: Janet