(a) what maximum force can be exerted by the string on the 10.0-kg block without causing the 5.00-kg block to slip? (b) use the system approach to calculate the

a block of mass m = 5.00 kg rides on top of a second block of mass m = 10.0 kg. a person attaches a string to the bottom block and pulls the system horizontally across a frictionless surface, as in figure below. friction between the two blocks keeps the 5.00-kg block from slipping off. if the coefficient of static friction is 0.350, (a) what maximum force can be exerted by the string on the 10.0-kg block without causing the 5.00-kg block to slip? (b) use the system approach to calculate the acceleration.

hi fikri!
 
this is one of those problems where i think doing the second part first might lead to a better understanding of what a system is, which seems to be a goal of the problem.
 
first, let's define a system that includes only the upper block.  defining a system allows us to define internal (totally inside the system) and external (from outside the system) forces.  only external forces affect the system's net acceleration, a, and are what we use in newton's second law (fnet = ma)
 
if the top block of mass m is not to slip, it must accelerate with the lower block in the horizontal direction.  for that, it needs a force acting on it in the horizontal direction.  what force could this be for the top block?  the only force that can act on the top block in the horizontal direction, if it is not slipping, is static friction (static, because the surfaces of the two blocks are not in motion with respect to each other).  remember that the string is only attached to the lower block, and thus does not apply a direct force to the upper block (our system).  that force is transmitted to the upper block through friction.
 
the expression for the static friction force is:  fs ≤ μsn
 
where
 
μs = coefficient of static friction
n = normal force
 
we are interested in the maximum possible force we can exert on the blocks without the top block slipping.  that means we are interested in the maximum static friction force, when fs = μsn
 
so let's set up newton's second law, in the horizontal direction (of motion), for our top block system in the case that we are exerting the maximum possible force without the top block slipping:
 
fnet-x = fs = ma --> μsn = ma --> (0.350)(n) = (5 kg)a --> a = (0.350)(n)/(5 kg)
 
this represents the maximum possible acceleration the top block can attain without slipping, and is the desired answer for part (b).  we can solve for that directly, once we have an expression for the normal force on the block (the normal force determines how tightly the surfaces in contact press together, which is part of what determines the magnitude of the friction force).
 
to solve for n, we have to look in the vertical direction, where n is.  in this direction, we have a normal force (up) on the top block, and gravity (mg) down.  so newton's second law in that direction becomes (with up defined as positive):
 
fnet-y = n - mg = ma = 0 (since we are assuming no net vertical acceleration on the upper block)
 
so n - mg = 0 --> n = mg = (5 kg)(9.8 m/s2) = 49 n
 
now,  a = (0.350)(49 n)/(5 kg) = 3.43 m/s2  [edit:  this confirms arturo's calculation above; not that it needed confirmation :) ]
 
this is the maximum possible (horizontal) acceleration of the top block without slipping, since it comes from the maximum value of the only force capable of accelerating the top block (system) horizontally.
 
********************************************
 
now, let's answer the first question second.
 
here is where redefining the system (which we are allowed to do at any point in a problem) can make things easier.  let's now define a system consisting of both the top and bottom blocks together (i sometimes refer to this as "wrapping our system bubble around," in this case, both blocks).  we can do this as long as we know that acceleration of the two block is the same, which is the situation we are dealing with here where the blocks move together.
 
by doing this, we simplify the horizontal direction for this problem.  with both blocks inside the system, the only horizontal external force acting on the system -- and thus the only force we have to put in newton's second law in the horizontal direction -- is the pulling force of the string (fp).  the frictional force between the blocks is internal, and can be left out of newton's second law.  in this case, newton's second law in the horizontal becomes:
 
fnet-x = fp = msysa
 
and we know that a has to be the maximum acceleration we already calculated (so that the upper block does not slip).  the mass of the system (msys) is just the total mass inside the system [note:  i did not make a point of calling it msys when the upper block alone was the system, since msys = m in that case].  since the system is both blocks, msys = (m+m) = (5 kg + 10 kg) = 15 kg.
 
so, the previous equation becomes:
 
fp = (15 kg)(3.43 m/s2) = 51.45 n  [edit: again, confirming arturo's result]
 
this is thus the maximum force we could pull on the string and not have the blocks slip.
 
in problems like this, i have always found it helpful to define a system that makes newton's second law as simple as possible, while still making the force(s) associated with the quantity i am trying to solve for external.
 
hope this helps!  it is just another way to look at the problem.

for the following solution, the effects of torques caused by the physical construction of the system (from the location of the attachment point of the string, or as caused by the height of the 10kg block, etc.) are presumed to be negligible.  this is likely to have been the intent of the problem, unless specific physical dimensions for the blocks and the location of the string attachment were provided in a separate diagram.
 
the force of gravity of the 5kg block on the 10kg block is fg = -mg = -5*9.81 = -49.05[n].
 
the normal force on the 5kg block is, in this case, equal and opposite to the force of gravity, such that fn = -fg = 49.05 [n].
 
the static friction restraining force, fs = mu\_s*fn = 0.350*49.05 = 17.17[n].
 
the 5kg block will remain at rest with respect to the 10kg block (which, in this case, means standing still on top of the 10kg block) as long as the acceleration of the 5kg block matches the acceleration of the 10kg block which will be as long as less than 17.17[n] of force is exerted on the 5kg block by the 10kg block.
 
while the blocks are held together by friction, the force exerted by the string accelerates the system of the two blocks according to the equation: 
 
f\_string\_max = f\_5kg + f\_10kg = 5*a + 10*a 
 
based on the static friction restraining force, the 5kg block starts to slip when f\_5kg is greater than 17.17[n].  therefore, the necessary acceleration, a = 17.17/5 = 3.434[m/s^2]
 
plugging this acceleration into the equation for the force on the string, we get:
 
f\_string\_max = 17.17 + 10*3.434 = 51.51[n]
 
thus the answer to part (a) is 51.5[n] and the answer to part (b) was already determined to be 3.43 [m/s^2].
 
 
 
 

without having the diagram, here is how i interpret the given information:
 
net external force in horizontal direction on m + m:
 
f = (m + m) a ⇒ a = f / (m + m)
 
now look at net horizontal force on block m by itself, with block m having the same acceleration as (m + m) (i.e. no slip):
 
ma = μsmg ⇒ a = μsg
 
then 
 
f / (m + m) = μsg ⇒ f = μs(m + m)g
 
this will be the maximum f that ensures no slipping of block m over block m.
 
(a)  
 
fmax =  μs(m + m)g = 0.350(5 + 10)(9.8) n = 51.45 n
 
(b)
 
a = fmax / (m + m) = 51.45 / (5 + 10) m/s2 = 3.43 m/s2